sco1 / dropmate-py

Python helpers for the EDC Dropmate

Hi there, I'm sco1! 👋

I'm a Boston-based Aerospace Engineer focused on test & evaluation.

Programming Interests

• Data collection, postprocessing, and visualization
• CLI and GUI tooling
• Code quality
• Software testing

I also find joy in making hilarious things like:

• randomly-change-self - Change all instances of "self" to "this" in one randomly chosen scope
• pytest-pahrametahrize - Parametrize your tests with a Boston accent
• brie-commit - A collection of cheesy pre-commit hooks
• pre-commit-fail-randomly - Randomly fail a pre-commit run

dropmate-py's Issues

Add audit for short time deltas between adjacent drops

Some instances of getting multiple log records for within the same drop have been noticed with short delays between the end of the "first" jump and beginning of the "second" jump. Starting with a simple delta around 15-20 minutes should be sufficient for the cases noticed so far

e.g.

2022-Jan-12T04-25-33Z,2022-Dec-12T04-32-54Z
2022-Jan-12T04-40-42Z,2022-Dec-12T04-41-59Z

and

2023-04-20T11:36:30Z,2023-04-20T11:37:05Z
2023-04-20T11:38:22Z,2023-04-20T11:39:04Z

Audit for poor health checks

e.g.

0,abc123,Good,poor,5.0,true,true,11,0,11,4,2022-04-20T20:10:08Z,2022-04-20T20:18:46Z,10039,5479,2023-04-20T21:58:19Z,2023-04-20T20:57:00.103Z,SM G970U,31,1.5.16

edit: Turns out we're not checking for battery health either

Support empty drop record

Dropmates without any log records will output columns with "na" values, e.g.:

0,ABC123,good,good,5.1,on,on,0,0,0,na,na,na,na,na,2023-07-13T14:47:31Z,2023-07-13T17:56:55Z

This will currently cause the parsing pipeline to error out. We can add support for this once we can get our hands on an empty record with the current app version.

edit: here's a sample empty log

serial_number,uid,battery,device_health,firmware_version,log_timestamp,log_altitude,total_flights,flights_over_18kft,recorded_flights,flight_index,start_time_utc,end_time_utc,start_barometric_altitude_msl_ft,end_barometric_altitude_msl_ft,dropmate_internal_time_utc,last_scanned_time_utc,scan_device_type,scan_device_os,dropmate_app_version
0,e123456067a65241,good,good,5.1,on,on,0,0,0,na,na,na,na,na,2023-04-20T13:02:41Z,2023-04-20T17:49:09Z,iPhone 14 Pro Max,16.6,1.4

Merge repeat drop logs during batch processing

Since the Dropmate retains all drop logs on-device, repeated scans of the same device is going to yield drop logs that contain records that are duplicated by earlier scans. Rather than yield repeated error(s) for the same drop record during batch auditing, it would be helpful to merge the files together and drop repeat records prior to performing the audit.

Convert audit errors to actual exceptions

Currently audit flags are stored as strings, but as more are potentially added it would be helpful to have the ability to filter them without having to parse for prefixes. Let's start by using exceptions instead.

Support legacy consolidated log file structure

Current functionality assumes that log records have been provided by Dropmate app version 1.5.16 or newer. Prior versions may not contain all the necessary data columns to conduct the data audit, and there may also be column naming discrepancies between the iOS and Android apps.

Some of the legacy layouts still may contain sufficient information to conduct the desired audits, so it might be possible to support these without significant effort. Rather than adjusting the current pipelines, it may be a nicer option to branch based on the header contents.

Here are some sample headers:

serial_number,uid,battery,log_timestamp,log_altitude,total_flights,prior_flights,flights_over_18kft,recorded_flights,flight_index,start_time_utc,end_time_utc,start_barometric_altitude_msl_ft,end_barometric_altitude_msl_ft

serial_number,uid,battery,log_timestamp,log_altitude,total_flights,flights_over_18kft,recorded_flights,flight_index,start_time_utc,end_time_utc,start_barometric_altitude_msl_ft,end_barometric_altitude_msl_ft,current_device_time_local

serial_number,uid,battery,device_health, firmware_version,log_timestamp,log_altitude,total_flights,flights_over_18kft,recorded_flights,flight_index,start_time_utc,end_time_utc,start_barometric_altitude_msl_ft,end_barometric_altitude_msl_ft,dropmate_internal_time,last_scanned_time