rogerlew / undaqtools

Unofficial NADS DAQ Toolkit

undaqTools [uhn-dak-tools]

undaqTools is the unofficial pythonic interface to the National Advanced Driving Simulator (NADS) Data AcQuisition (DAQ) files (i.e. NADS miniSim (TM) datafiles).

undaqTools (unofficial) vs. ndaqTools (official)

Approach:

ndaqTools focuses on ease-of-use (via GUIs) and templating. In contrast, the focus of undaqTools is on data abstraction and simplification of data extraction.

Programming Language:

ndaqTools is built on MATLAB. Whereas, undaqTools utilizes the opensource Python programming language (as well as Python's opensource scientific stack (NumPy, SciPy, Matplotlib, and h5py).

Data Storage:

ndaqTools stores and loads DAQ data as .mat files. undaqTools stores and loads data in HDF5 containers.

Summary

undaqTools converts the native DAQ data into the HDF5 high performance binary data format. From there data can be quickly accessed in Python using the Daq class of undaqTools. Daq objects are dict subclasses. The keys are the elements of the daq and the values are stored in a numpy array subclass called Element. Each Element contain measure relevent metadata stored as instance attributes so these metadata does not have to be accessed across datastructures.

Each Element object also has a frames attribute that keeps track of the frames cooresponding to the element. This means that when they are sliced the resulting views keeps the frames in sync without any additional book-keeping to the end user! The element data can also be sliced and indexed using frame indices. These abstractions simplify the complexity of managing these complex datasets (see class hierarchy diagram below).

undaqTools can also unpack and process dynamic objects. Each dynamic object becomes a DynObj instance in the Daq's dynobj dict attribute. The keys to the dynamic objects are the dynamic object names. The DynObj containers store the measure that you would expect (position, frames, speed, heading, etc.) as numpy.array objects as well as a running relative distance measure to the Own Vehicle.

These abstractions from the original DAQ representation are intended to provide the end user the convenience and capability to decide how to best utilize their data.

Simplified Class Hierarchy Diagram:

Daq(dict)
  + [elemName] -> Element (ndarray)
  |                 + data
  |                 + frames (ndarray)
  |                 + rate
  |                 + type
  |                 + ...
  |               --------------------
  |                 + isCSSDC()
  |
  + dynobjs (dict)
  |   + [doName] -> DynObj (object)
  |                   + pos (Element)
  |                   + heading (Element)
  |                   + speed (Element)
  |                   + ...
  |                 ---------------------
  |                   + process()
  |                   + read_hd5()
  |                   + write_hd5()
  |                   + ...
  |
  + info (numedTuple)
  |   + run
  |   + subject
  |   + date
  |   + ...
  |
  | + etc (dict)
  | + ...
--------------------------------------------
  | + read_daq() <==> read()
  | + read_hd5py()
  | + write_hd5py()
  | + write_mat()
  | ...

Source

https://github.com/rogerlew/undaqTools

Release

https://pypi.python.org/pypi/undaqTools/

Documentation

http://pythonhosted.org/undaqTools/

Acknowledgments

Daq.read_daq() method based on Chris Schwarz and others of NADS convert_daq.py script.

undaqtools's Issues

Error

Hi Roger,

Thank you for your awesome package. I am trying to read in a DAQ file since the automation updates to the simulator. But some, not all, files produce an error. I am not sure if it is because something is outdated in undaqTools. Could you please have a look? Thank you.

DAQ file

I have 2 DAQ files that I am trying to read in. The WithTraffic file is successfully read in, but the NoTraffic file produces the error below. You can download both files here

Code and Error

I have the following script that I run with python 02_convert_daq_files_to_hdf5.py. But it produces the error below.

# Changing directory to the folder where DAQ files reside ----------------------------
import os
os.chdir("C:\\Users\\HP\\Desktop\\driving_sim\\data\\daq")



# Getting the names of all DAQ files
import glob
ctnames = glob.glob('*.daq')
#print(ctnames)




# Importing and using undaq tools to convert all DAQ files to HDF5 format
# import sys

from undaqTools import Daq
for n in ctnames:
  daq = Daq()
  x = daq.read(n)
  daq.write_hd5(x)
  # print(type(x))

Traceback (most recent call last):
File "02_convert_daq_files_to_hdf5.py", line 30, in
x = daq.read(n)
File "C:\Python27\lib\site-packages\undaqTools\daq.py", line 295, in read_daq
self._interpolate_missing_frames()
File "C:\Python27\lib\site-packages\undaqTools\daq.py", line 559, in _interpolate_missing_frames
for j in xrange(elem.numvalues)]
File "C:\Python27\lib\site-packages\numpy\lib\function_base.py", line 1412, in interp
return interp_func(x, xp, fp, left, right)
TypeError: Cannot cast array data from dtype('S1') to dtype('float64') according to the rule 'safe'

Any plan to release the package for python 3?

I am trying to make your awesome tool undaqTools accessible to R language users by wrapping Daq(). However, python 2 support is going to end in R (via reticulate package). Do you have any plans to release the package for python 3?