asukhanov / apstrim

Logger and extractor of time-series data.

apstrim

Logger and extractor of time-series data (e.g. EPICS PVs).

• Data are objects, indexed in time order.
• Supported Control Infrastructures: EPICS, ADO, LITE. Easy extendable.
• Wide range of data objects: strings, lists, maps, numpy arrays, custom.
• Data objects could be inhomogeneous and have arbitrary frequency.
• Self-describing data format, no schema required.
• Data objects are binary-serialized using MessagePack.
• Fast online compression.
• Fast random-access retrieval of data objects for selected time interval.
• Simultaneous writing and reading.
• Typical speed of compressed serialization to a logbook file is 80 MB/s.
• De-serialization speed is up to 1200 MB/s when the logbook is cached in memory.
• Basic plotting of the logged data.

Installation

Dependencies: msgpack, caproto, lz4framed. These packages will be installed using pip:

pip3 install apstrim

The example program for deserialization and plotting apstrim.view, requires additional package: pyqtgraph.

API refrerence

apstrim

scan

Examples

Serialization

# Serialization of one float64 parameter from an EPICS simulated scope IOC:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV
Logging finished for 1 sections, 1 parLists, 7.263 KB.
...

# The same with compression:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV --compress
Logging finished for 1 sections, 1 parLists, 6.101 KB. Compression ratio:1.19
...

# Serialization 1000-element array and one scalar of floats:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV,Waveform_RBV --compress
Logging finished for 1 sections, 2 parLists, 2405.354 KB. Compression ratio:1.0
...
# Note, Compression is poor for floating point arrays with high entropy.

# Serialization of an incrementing integer parameter:
:python -m apstrim -nLITE --compress liteHost:dev1:cycle
Logging finished for 1 sections, 1 parLists, 56.526 KB. Compression ratio:1.25
...
# In this case the normalized compressed volume is 9.3 bytes per entry.
# Each entry consist of an int64 timestamp and an int64 value, which would 
# occupy 16 bytes per entry using standard writing.

De-serialization

Example of deserialization and plotting of all parameters from several logbooks.

python -m apstrim.view -i all -p *.aps

Python code snippet to extract items 1,2 and 3 from a logbook for 20 seconds interval starting on 2021-08-12 at 23:31:31.

from apstrim.scan import APScan
apscan = APScan('aLogbook.aps')
headers = apscan.get_headers()
print(f'{headers["Index"]}')
extracted = apscan.extract_objects(span=20, items=[1,2,3], startTime='210812_233131')
print(f'{extracted[3]}')# print the extracted data for item[3]
# returned:
{'par': 'liteBridge.peakSimulator:rps',           # object (PV) name of the item[3]
'times': [1628825500.8938403, 1628825510.898658], # list of the item[3] timestamps
'values': [95.675125, 95.55396]}                  # list of the item[3] values