clydemcqueen / ardusub_log_tools Goto Github PK

Some BIN files from Nereo are named with time/date, but some are not. (Why?)

BIN_info.py: scan a BIN file for GPS records and report on the time/date information.

Summary: the line os.environ['MAVLINK20'] = '1' isn't doing anything, and it should be removed.

Longer story:

To set the MAVLink wire protocol to 2.0 you need to set os.environ['MAVLINK20'] = '1' before importing pymavlink. Setting the wire protocol only matters when writing (packing) new messages; when reading (unpacking) the wire protocol is determined by the magic byte (0xFE vs 0xFD) in the message. None of the tools write new messages, so we don't need this line, and probably don't need the import os as well.

MAVLink version reference

tlog_info.py doesn't use table_types.py, while tlog_merge does. Consolidate this logic.

By default wl_ugps_logger points to demo.waterlinked.com. The default should be the typical field setup: 192.168.2.194.

As we prepare to use the WL UGPS external bridge to send NMEA messages to the WL G2 box, it will be helpful to add a few more endpoints to the wl_ugps_logger:

• Global position of G2 box: /api/v1/position/master
• Global position of U1 locator: /api/v1/position/global

Write parameter files that MAVProxy will read:

• output file name should be *.parm
• line should be , where value is float or int

Add a few tests and automagically run them on each push or PR

*.BIN doesn't do the right thing

A dive may generate multiple telemetry (tlog) and dataflash (BIN) logs. These can be organized in a directory structure, with all of the log files for a dive sitting in a directory.

Create a dive.py tool that:

• opens all of the tlog and BIN files in a directory
• uses time information in the files (e.g., SYSTEM_TIME MAVLink messages) to build a single timeline that spans all of the telemetry and dataflash logs. The names of the files may also contain some useful information, though it may be less trustworthy than the data contained in the files.
• report any errors, such as missing log files
• print interesting information about the dive

The timeline generator should be in a separate Python file so that it can be used by other tools. E.g., merge.py may replace tlog_merge.py and BIN_merge.py and merge data from multiple files and log types.

• re-arrange as a module
• add setup.py and other files
• publish on PyPi

Move wl_* scripts to wl_ugps_acoustic_analysis

Consider creating a BlueOS extension to replace the wl_* scripts

Add a new tlog_explode.py ... tool (wrapper) that implements tlog_merge.py --no-merge --explode ....

Add a new segment.py tool that will do the following:

• Open one or more tlog files from the same dive and logically stitch them together in time order.
• Segment the messages by time, where segments alternate between discard and keep.
• Write the keep segments to new, smaller tlog files.

The target use case is a survey that spans several hours, where a single tlog file spans several transects. We want to easily pull out the transects for further processing.

Proposed parameters:

• [--timestamps xxx,yyy,zzz] A comma-separated (no spaces) list of timestamps . If no timestamps are provided, the tool will stitch tlog files together, but won't segment them.
• [--keep-first] By default, the messages before the 1st timestamp in --timestamps are considered a discard segment. Use this option if the first segment is a keep segment instead.
• [--types aaa,bbb,ccc] A comma-separated (no spaces) list of message types to write. If none are provided, all types are included.
• path ... One or more tlog files. If there is more than 1 tlog file they will be stitched together in the order provided.

For example, this command will break foo.tlog into 5 segments at 100-second boundaries. The segments between 100-200s and 300-400s will be written as new tlog files:

segment.py --timestamps 100,200,300,400 foo.tlog

Same as above, but only the DISTANCE_SENSOR messages are written in the new tlog files:

segment.py --timestamps 100,200,300,400 --types DISTANCE_SENSOR foo.tlog

Timestamps can be specified in 2 ways:

• Seconds since the start of the earliest tlog file in the list, e.g., 100.
• Unix time (seconds since January 1st, 1970 UTC), e.g., 1694633807. Any timestamp > 10000000 is treated as Unix timestamp.

Limitations:

• The user must provide the timestamps by examining the files.
• This only works on tlog files, not BIN (Dataflash) files.

See #22 for a more general tool that re-constructs a dive using both BIN and tlog files. This is more complicated because BIN files use a different clock.

Add comments (trailing # Comment strings) to the EK3 parameters to make it easy to see how sensors are fused.

tlog_param.py writes all values as floats. It should write an int value if the type is some sort of int.