vtools.datastore package

Submodules

vtools.datastore.catalog module

class Catalog

Bases: object

Class contain a list of CatalogEntry, also provide a number of operations on those entries.

copy(entry)

Copy data associated with entry1 into entry2.

data_references(selector=None, extent=None)

return a itertor of the datareferences to paths based on selector.

entries()

return a iterator over the inside list of CatalogEntries.

filter_catalog(selector)

Filter the current catalog based on given selector, return a list of data references conforms to selector. Usually selector is a line of regular expression string, such as for a xxx data source, it may be “A=RLTM;B=RSAC*;”.

get_data_reference(entry, extent=None)

return data record reference on a given catalog entry.

load_entry_full(index)

Load the catalog entry by index in full, this sub used in lazy loading mode.

modify(old_entry, new_entry)

Modify a entry within Catalog.

reload()

Lazily reload catalog info when entry has changed.

remove(entry)

Remove a entry from Catalog.

schema()

return the schema which defined this catalog

class CatalogEntry(schema, catalog=None)

Bases: object

Single entry of a catalog of a datasource

add_dimension_scale(dimension)

Function to add a dimension instance to dimension list.

add_item(name, value)

Function to set the value of metadata within a CatalogEntry instance with the name specified.

dimension_scales()

Function to retrieve all the dimension defined within this CatalogEntry.

get_index()
has_item(name)

Check if a item exist within a catalog entry.

index

index of entry within a catalog

item(name)

Function to return the value of metadata item within a CatalogEntry instance with the name specified

If the item specified actually doesn’t exist, error will be raised

item_names()

Return names of all meta data item stored in this entry.

load_full()

Load full record about a entry if applicable.

set_index(value)

Set inner index value.

set_item(name, value)

Function set a item value. It actually call add_item.

catalog_string(c, header=True)

Return a string representation of a catalog

vtools.datastore.catalog_schema module

class CatalogSchemaItem(attr_dict=None)

Bases: object

Class stores the definition for catalog entry item.

append_element(element_name, value)

This function append a definition element to exist list.

element_name is the string name of definition, value is the context of the definition, this function preclude user to reset any existing element values.

get_element(element_name)

This function return a definition element based on name given.

element_name is the string name of definition.

has_element(element_name)

Function to check if element_name exist in this schema item

vtools.datastore.catalog_service module

class CatalogService

Bases: object

abstract class defined interface of retrieving catalog from data source.

get_catalog(source)

Retrieve Catalog from data source specified by argument source. Single catalog object or list of catalog will be returned.

identification = ''

vtools.datastore.data_access_service module

exception DataAccessError

Bases: exceptions.Exception

class DataAccessService

Bases: object

abstract class defined interface of retrieveing data from data source.

add_data(dataref, data)

Add the time series data to place specified by dataref.

assemble(translation, **kargs)

Make a data ref based on the mapper and input used in transfer majorly.

dissect(dataref)

Analyze the elements contained within a dataref and return result as dict, acutall result depends on data service.

get_data(dataref, overlap=None)

Retrieve the time series specified by argument dataref.

modify_data(dataref, data)

Modify the time series specified by argument dataref.

remove_data(dataref)

Remove the time series specified by argument dataref.

vtools.datastore.data_reference module

class DataReference(id, source=None, view=None, selector=None, extent=None)

Bases: object

Class maintain a reference a time series in plain language.

extents()

Parse extent string and return a list of tuples contain each exent element’s name and value in strings

DataReferenceFactory(id, source=None, view=None, selector=None, extent=None)

vtools.datastore.data_service_manager module

class DataServiceManager

Bases: vtools.datastore.data_service_manager.Singleton

get_service(idd)
class Singleton

Bases: object

vtools.datastore.dimension module

class ChannelDimension(channel_list=[])

Bases: vtools.datastore.dimension.Dimension

Sub Class store meaningful index information of a dataset dimension as a list.

get_channel(index)

Return data channel at index

get_index(channel)

return the index of a channel in the inner list.

set_channels(channel_list)

Set internal data channel list.

class Dimension

Bases: object

Parent Class store meaningful index information of a dataset dimension.

For instance, within a multi-dimensional array dataset, first dimension is indexing of location, second dimension is indexing of time, and so on.

label = 'unkown_scale'
name = ''
class RangeDimension(range=None, label='time_window', name='time_window', index=0, interval=None)

Bases: vtools.datastore.dimension.Dimension

Sub Class store meaningful index information of a dataset dimension as a pair of start and end of some kind of value.

get_range()

Return range setted

isregular()
set_range(range)

Setup inner range of range dimension

vtools.datastore.group module

group_by_nan(data, compress_size)
ts_group(ts, gap_size=1000)

Split a time sereies with gaps to a group time series with gap large then nan_size remvoed.

input:
ts: instance of timeseries, regular or irregular. gap_size: integer specify the size of gap to be splitted
output:
A number of time series

vtools.datastore.optionparse module

Author:
  1. Simionato
Date:

April 2004
Title:

A much simplified interface to optparse.

You should use optionparse in your scripts as follows. First, write a module level docstring containing something like this (this is just an example):

‘’‘usage: %prog files [options]
-d, –delete: delete all files -e, –erase = ERASE: erase the given file’‘’

Then write a main program of this kind:

# sketch of a script to delete files if __name__==’__main__’:

import optionparse option,args=optionparse.parse(__doc__) if not args and not option: optionparse.exit() elif option.delete: print “Delete all files” elif option.erase: print “Delete the given file”

Notice that optionparse parses the docstring by looking at the characters “,”, “:”, “=”, “n”, so be careful in using them. If the docstring is not correctly formatted you will get a SyntaxError or worse, the script will not work as expected.

exception ParsingError

Bases: exceptions.Exception

exit(msg='')
nonzero(self)

True if options were given

parse(docstring, arglist=None)

vtools.datastore.service module

class Register(name, bases, dict)

Bases: type

A tiny metaclass to help classes register themselves automagically

class Service

Bases: vtools.datastore.data_access_service.DataAccessService, vtools.datastore.catalog_service.CatalogService

all_service = {}
static factory(identification)
identification = 'vtools.datastore.ui.service'
static register(name, cls)
get_data(catalog_entry)

vtools.datastore.test_group module

class TestGroup(methodName='runTest')

Bases: unittest.case.TestCase

test functionality of shift operations

test_group_by_nan()
test_ts_group()

vtools.datastore.transfer module

Transfer and transform data from one source to another This module provides functionality for opening one data source,grabbing data that matches selecting criteria, transforming it or clipping it in time and then exporting it to another data source.

usage: transfer [options]
-i, –in=input: data store of orginal data records. -s, –selector=select: selection criteria of input data. -o, –out=output: data store where data will be written. -d, –dest=dest: optional,map from selector to the output location in the output data store. -t, –trans=transformer: optional,path to the tranformation func to be applied. -e, –extent=extent: optional, extent within a selected time series. -u, –usage: help info.
transfer(s1, reference1, s2, reference2, transform=None, **func_args)

Transfer time series data pointed by reference1 to location refered by reference2, with optional transformation operation on orginal data series.

input:

reference1: instance/list of data reference class of orginal data. reference2: instance/list of data reference class of new data. transformfunc: transforming operation function,can be as full path

string of existing function provided by vtools package, ,etc. “vtools.functions.data_interpolate.interpolate_its _rts” or a function object defined by user.

**func_args: arguments dic for transform function.

output:
None.
batch_transfer(source, dest=None, selector=None, extent=None, mapper=None, transform=None, **func_args)

Batch transfer selected time series from data source1 to source2, with optional transformation operation on orginal data series.

usage: batch_transfer(source1,selector1,source2,selector2,transform,**func_args): input:

source: path to source (full path or relative path). dest: path to destination of transfer (full path or relative path). selector: selection criteria for source1, given as string. extent: data extent selection for source1. mapper: selection seting for source2, given as string, it

is translation for the moment(thus no support for transfer to exsiting paths in exisiting data source available yet).

optional input:

transform: transforming operation function,can be as full path
string of existing function provided by vtools package, ,etc. “vtools.functions.data_interpolate.interpolate_its _rts” or a function object defined by user.

**func_args: arguments dic for transform function.

output:
None.

vtools.datastore.translate module

Functions to translate between different data references.

translate_references(s1, references, s2, destination, translation=None)

Translate a number of datareferences (which usually point to the location of a different data source, for the moment same type of data source assumed) to the references points to the locations within the current data source (one to one).

input:
s1: service for input references. references: list of data references. s2: service for input destination. destination: full path to detination data source,string. scenario of translation:
output:
a list of new references.
To do:
Implement translation between different data source.

Module contents