SELKIELogger.SLFiles.DatFile Class Reference

Represent a SELKIELogger data file and associated common operations. More...

Public Member Functions
def	__init__ (self, filename, pcs=IDs.SLSOURCE_TIMER)
	Create DatFile instance. More...
def	addSourceMap (self, sm)
	Associate an SLChannelMap instance with this file. More...
def	prepConverters (self, force=False, includeTS=False)
	Generate and cache functions to convert each channel into defined fields and corresponding field names suitable for creating a DataFrame later. More...
def	buildRecord (self, msgStack)
	Convert a group (stack) of messages into a single record, keyed by field names. More...
def	messages (self, source=None, channel=None)
	Process data file and yield messages, optionally restricted to those matching a specific source and/or channel ID. More...
def	processMessages (self, includeTS=False, force=False, chunkSize=100000)
	Process messages and return. More...
def	yieldDataFrame (self, dropna=False, resample=None, convertEpoch=False)
	Process file and yield results as dataframes that can be merged later. More...
def	asDataFrame (self, dropna=False, resample=None, convertEpoch=False)
	Wrapper around yieldDataFrame. More...

Static Public Member Functions
def	tryParse (value)
	Attempt to convert value to float. More...

Private Attributes
	_fn
	File name and path.
	_fields
	Cached conversion functions.
	_pcs
	Primary Clock Source ID.
	_sm
	Source/Channel Map.
	_records
	File data records (once parsed)
	_columnList

Detailed Description

Represent a SELKIELogger data file and associated common operations.

Definition at line 106 of file SLFiles.py.

Constructor & Destructor Documentation

__init__()

def SELKIELogger.SLFiles.DatFile.__init__	(		self,
			filename,
			pcs = IDs.SLSOURCE_TIMER
	)

Create DatFile instance.

Does not open or parse file.

Parameters

filename	File name and path
pcs	Primary Clock Source (Source ID)

Definition at line 111 of file SLFiles.py.

Member Function Documentation

addSourceMap()

def SELKIELogger.SLFiles.DatFile.addSourceMap	(		self,
			sm
	)

Associate an SLChannelMap instance with this file.

Used to map source and channel IDs to names.

Parameters

sm	SLChannelMap (see VarFile.getSourceMap)

Returns

None

Definition at line 132 of file SLFiles.py.

asDataFrame()

def SELKIELogger.SLFiles.DatFile.asDataFrame	(		self,
			dropna = False,
			resample = None,
			convertEpoch = False
	)

Wrapper around yieldDataFrame.

Processes all records and merges them into a single frame.

Parameters

dropna

Drop empty records.

See also

yieldDataFrame()

Parameters

resample

Resampling interval.

See also

yieldDataFrame()

Returns

pandas.DataFrame containing file data

Definition at line 524 of file SLFiles.py.

buildRecord()

def SELKIELogger.SLFiles.DatFile.buildRecord	(		self,
			msgStack
	)

Convert a group (stack) of messages into a single record, keyed by field names.

Field values default to None, and in the event that multiple values for a single field are received, the last value received will be stored.

Parameters

msgStack

Group of messages received for a specific interval

Returns

Record/dictionary of values keyed by field name

Definition at line 343 of file SLFiles.py.

messages()

def SELKIELogger.SLFiles.DatFile.messages	(		self,
			source = None,
			channel = None
	)

Process data file and yield messages, optionally restricted to those matching a specific source and/or channel ID.

• x.messages() - Yields all messages
• x.messages(source=0x10) - Yields all messages from source 0x10 (GPS0)
• x.messages(channel=0x03) - Yields all channel 3 (raw) messages from any source
• x.messages(0x10, 0x03) - Yield all channel 3 messages from source 0x10

Parameters

source	Optional: Source ID to match
channel	Optional: Channel ID to match

Returns

Yields messages in file order

Definition at line 370 of file SLFiles.py.

prepConverters()

def SELKIELogger.SLFiles.DatFile.prepConverters	(		self,
			force = False,
			includeTS = False
	)

Generate and cache functions to convert each channel into defined fields and corresponding field names suitable for creating a DataFrame later.

Names and functions are returned as a collection keyed by source and channel. As each message may produce multiple message, names and functions are each returned as lists containing a minimum of one entry.

fields = x.prepConverters()
names, funcs = fields[source][channel]
out = {}
out[names[0]] = funcs[0](input)

Where input is an input SLMessage

Parameters

force	Regenerate fields rather than using cached values
includeTS	Retain timestamp field as data column as well as index

Returns

Collection of conversion functions and field names

Definition at line 167 of file SLFiles.py.

processMessages()

def SELKIELogger.SLFiles.DatFile.processMessages	(		self,
			includeTS = False,
			force = False,
			chunkSize = 100000
	)

Process messages and return.

Will yield data in chunks.

Parameters

includeTS	Passed to prepConverters()
force	Passed to prepConverters()
chunkSize	Yield records after this many timestamps

Returns

List of tuples containing timestamp and dictionary of records

Definition at line 406 of file SLFiles.py.

tryParse()

def SELKIELogger.SLFiles.DatFile.tryParse (   value )

static

Attempt to convert value to float.

If unsuccessful, parse as JSON and attempt to return a float from either a) the sole value within the object or b) the data associated with a key named "value".

Parameters

value

Input value

Returns

Floating point value or np.nan on failure

Definition at line 144 of file SLFiles.py.

yieldDataFrame()

def SELKIELogger.SLFiles.DatFile.yieldDataFrame	(		self,
			dropna = False,
			resample = None,
			convertEpoch = False
	)

Process file and yield results as dataframes that can be merged later.

Optionally drop empty records and perform naive averaging over a given resample interval.

Parameters

dropna	Drop rows consisting entirely of NaN/None vales
resample	Resampling interval

Returns

pandas.DataFrame representing a chunk of file data

Definition at line 461 of file SLFiles.py.

---

The documentation for this class was generated from the following file:

• /home/runner/work/Logger/Logger/python/SELKIELogger/SLFiles.py