cleanflight / blackbox-tools

Tools for working with blackbox flight logs

Blackbox flight data recorder tools

Introduction

These tools allow you to convert flight data logs recorded by Cleanflight's Blackbox feature into CSV files (comma-separated values) for analysis, or into a series of PNG files which you could turn into a video.

You can download the latest executable versions of these tools for Mac or Windows from the "releases" tab above. If you're running Linux, you must build the tools from source (instructions are further down this page).

Using the blackbox_decode tool

This tool converts a flight log binary ".TXT" file into CSV format. Typical usage (from the command line) would be like:

blackbox_decode LOG00001.TXT

That'll decode the log to LOG00001.01.csv and print out some statistics about the log. If you're using Windows, you can drag and drop your log files onto blackbox_decode and they'll all be decoded. Please note that you shouldn't discard the original ".TXT" file, because it is required as input for other tools like the PNG image renderer.

If your log file contains GPS data then a ".gpx" file will also be produced. This file can be opened in Google Earth or some other GPS mapping software for analysis. This feature is experimental.

Use the --help option to show more details:

Blackbox flight log decoder by Nicholas Sherlock

Usage:
     blackbox_decode [options] <input logs>

Options:
   --help                   This page
   --index <num>            Choose the log from the file that should be decoded (or omit to decode all)
   --limits                 Print the limits and range of each field
   --stdout                 Write log to stdout instead of to a file
   --unit-amperage <unit>   Current meter unit (raw|mA|A), default is A (amps)
   --unit-frame-time <unit> Frame timestamp unit (us|s), default is us (microseconds)
   --unit-height <unit>     Height unit (m|cm|ft), default is cm (centimeters)
   --unit-rotation <unit>   Rate of rotation unit (raw|deg/s|rad/s), default is raw
   --unit-acceleration <u>  Acceleration unit (raw|g|m/s2), default is raw
   --unit-gps-speed <unit>  GPS speed unit (mps|kph|mph), default is mps (meters per second)
   --unit-vbat <unit>       Vbat unit (raw|mV|V), default is V (volts)
   --merge-gps              Merge GPS data into the main CSV log file instead of writing it separately
   --simulate-current-meter Simulate a virtual current meter using throttle data
   --sim-current-meter-scale   Override the FC's settings for the current meter simulation
   --sim-current-meter-offset  Override the FC's settings for the current meter simulation
   --simulate-imu           Compute tilt/roll/heading fields from gyro/accel/mag data
   --imu-ignore-mag         Ignore magnetometer data when computing heading
   --declination <val>      Set magnetic declination in degrees.minutes format (e.g. -12.58 for New York)
   --declination-dec <val>  Set magnetic declination in decimal degrees (e.g. -12.97 for New York)
   --debug                  Show extra debugging information
   --raw                    Don't apply predictions to fields (show raw field deltas)

Using the blackbox_render tool

This tool converts a flight log binary ".TXT" file into a series of transparent PNG images that you could overlay onto your flight video using a video editor (like DaVinci Resolve ). Typical usage (from the command line) would be like:

blackbox_render LOG00001.TXT

This will create PNG files at 30 fps into a new directory called LOG00001.01 next to the log file.

Use the --help option to show more details:

Blackbox flight log renderer by Nicholas Sherlock

Usage:
     blackbox_render [options] <logfilename.txt>

Options:
   --help                 This page
   --index <num>          Choose which log from the file should be rendered
   --width <px>           Choose the width of the image (default 1920)
   --height <px>          Choose the height of the image (default 1080)
   --fps                  FPS of the resulting video (default 30)
   --threads              Number of threads to use to render frames (default 3)
   --prefix <filename>    Set the prefix of the output frame filenames
   --start <x:xx>         Begin the log at this time offset (default 0:00)
   --end <x:xx>           End the log at this time offset
   --[no-]draw-pid-table  Show table with PIDs and gyros (default on)
   --[no-]draw-craft      Show craft drawing (default on)
   --[no-]draw-sticks     Show RC command sticks (default on)
   --[no-]draw-time       Show frame number and time in bottom right (default on)
   --[no-]draw-acc        Show accelerometer data and amperage in bottom left (default on)
   --[no-]plot-motor      Draw motors on the upper graph (default on)
   --[no-]plot-pid        Draw PIDs on the lower graph (default off)
   --[no-]plot-gyro       Draw gyroscopes on the lower graph (default on)
   --sticks-top <px>      Offset the stick overlay from the top (default off)
   --sticks-right <px>    Offset the stick overlay from the right (default off)
   --sticks-width <px>    Size of the stick area (default off)
   --sticks-radius <px>   Diameter of the sticks (default relative to image size)
   --sticks-trail-radius <px>  Diameter of the sticks (default same ad stick radius)
   --craft-top <px>       Offset the craft overlay from the top (default off)
   --craft-right <px>     Offset the craft overlay from the right (default off)
   --craft-width <px>     Size of the craft area (default off)
   --smoothing-pid <n>    Smoothing window for the PIDs (default 4)
   --smoothing-gyro <n>   Smoothing window for the gyroscopes (default 2)
   --smoothing-motor <n>  Smoothing window for the motors (default 2)
   --unit-gyro <raw|degree>  Unit for the gyro values in the table (default raw)
   --prop-style <name>    Style of propeller display (pie/blades, default pie)
   --gapless              Fill in gaps in the log with straight lines
   --raw-amperage         Print the current sensor ADC value along with computed amperage
   --sticks-text-color    Set the RGBA text color (default 1.0,1.0,1.0,1.0)
   --sticks-color         Set the RGBA sticks color (default 1.0,0.4,0.4,1.0)
   --sticks-area-color    Set the RGBA sticks area color (default 0.3,0.3,0.3,0.8)
   --sticks-cross-color   Set the RGBA sticks crosshair color (default 0.75,0.75,0.75,0.5)
   --sticks-trail-length <px> Length of the stick trails (default 0)

(At least on Windows) if you just want to render a log file using the defaults, you can drag and drop a log onto the blackbox_render program and it'll start generating the PNGs immediately.

Assembling video with DaVinci Resolve

DaVinci Resolve is one free option for turning your PNGs into a video that is synced with your flight video.

In Resolve, create a new project. On the Media tab, find your flight video and drag it from the top pane to the bottom one to add it to your media library. When prompted, choose the option to change your framerate/resolution settings to match the video you added. Now add the PNG series that you rendered with blackbox_render (it'll be represented as a single file). If the PNGs don't appear in the media library, try right clicking on the folder that contains them and selecting "Refresh".

Now over on the Edit tab, click File -> New Timeline and click Ok. At the top left, switch over to the Media Pool. Drag and drop your flight video onto the timeline, then drag and drop the Blackbox stream onto the video track above it. You'll need to play around with the alignment between the two tracks to sync things up (I usually sync up the audio from the initial throttle-up with the throttle-up shown on the motor graph).

If your Blackbox PNGs were rendered using the default settings (30 FPS) and your flight video is 60 FPS, you'll need to right-click on the Blackbox stream and click "change clip speed". Enter 30 FPS since this is the FPS that you rendered the Blackbox PNGs at.

Save your viewers' ears by dragging the midpoint of the audio track downwards to reduce its volume, or replace the audio with your own music track!

Now on the Deliver tab, choose something like the "video sharing export" Easy Setup preset. On the left pane, choose an output folder for the "render to:" setting. Then click the "add job to render queue" button on the left. Now click the "start render" button on the right to begin rendering the output.

Building tools

If you just want to download some prebuilt versions of these tools, head to the "releases" tab on the GitHub page. However, if you want to build your own binaries, or you're on Linux where we haven't provided binaries, please read on.

The blackbox_decode tool for turning binary flight logs into CSV doesn't depend on any libraries, so can be built by running make obj/blackbox_decode. You can add the resulting obj/blackbox_decode program to your system path to make it easier to run.

The blackbox_render tool renders a binary flight log into a series of PNG images which you can overlay on your flight video. Please read the section below that most closely matches your operating system for instructions on getting the libcairo library required to build the blackbox_render tool.

Ubuntu

You can get the tools required for building by entering these commands into the terminal:

sudo apt-get update
sudo apt-get install make gcc libcairo2-dev

Build blackbox_render by running make obj/blackbox_render (or build both tools by just running make).

MacOSX

The easiest way to build is to install the Xcode development tool, then install an environment like Homebrew or MacPorts onto your system.

From MacPorts, you would do this to get LibCairo:

sudo port selfupdate
sudo port install cairo

Afterwards you can run make to build blackbox_render.

If you are using Homebrew instead of MacPorts, run:

brew install cairo --without-x11 pkg-config

Afterwards you can run make to build blackbox_render.

If you get an error "Package 'xcb-shm', required by 'cairo', not found", your installed version of Cairo depends on X11 but your Homebrew X11 libraries are not on your pkgconfig path, so the build process cannot find them. Try this to add them to your path:

$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig

Windows

The tools can be built with Visual Studio Express 2013, just open up the solution in the visual-studio/ folder. You'll need to include the .DLL files from lib/win32 in the same directory as your built executable.

License

This project is licensed under GPLv3.

The binary version of blackbox_render for MacOSX is statically linked to these libraries:

• libbz2 http://www.bzip.org/ (BSD-like)
• zlib http://www.zlib.net/
• libcairo & libpixman http://cairographics.org/ (LGPL)
• libfreetype http://www.freetype.org/ (BSD-like/GPLv2)
• libpng16 http://www.libpng.org/pub/png/libpng.html

The windows binary of blackbox_render additionally ships with these DLLs:

• libiconv https://www.gnu.org/software/libiconv/ (LGPL)
• libfontconfig http://www.freedesktop.org/wiki/Software/fontconfig/
• libxml2 http://xmlsoft.org/ (MIT)
• liblzma http://tukaani.org/xz/ (Public Domain)

This font is included with both binary and source distributions:

• Source Sans Pro - Regular https://github.com/adobe-fonts/source-sans-pro (SIL Open Font license)

Both binary and source builds include IMU code from Baseflight https://github.com/multiwii/baseflight (GPLv3)

blackbox-tools's Issues

blackbox_render fps parameter ignores fraction part

if one does call blacckbox_render with for example --fps=49.3 or --fps49,3 the result is the same as with --fps=49

could this be fixed? without this I need to make my video editing software create additional frames (or drop some if I instead would generate the blackbox pngs with 50 fps) which reduces the smoothness of the flow of the frames..

blackbox_render issue with large timestamps

I just tried the luxfloat PID controller and the blackbox-log of that flight can be decoded just fine, but the rendering shows a unchanging PID table, unchanging command inputs and missing all the lines of motor outputs and gyro.

blackbox_decode works just fine and produces a csv document that looks all right.

kaefert@mint ~/Downloads/quadcopter_stuff/blackbox-log/2015-08-09_1110 try luxfloat PID $ ~/Downloads/quadcopter_stuff/blackbox-log/blackbox-tools/obj/black
box_decode LOG00138.TXT 
Decoding log 'LOG00138.TXT' to 'LOG00138.01.csv'...

Log 1 of 1, start 43:28.366, end 45:53.195, duration 02:24.828

Statistics
Looptime           3504 avg           28.3 std dev (0.8%)
I frames    1292   51.2 bytes avg    66199 bytes total
P frames   40029   29.4 bytes avg  1175762 bytes total
H frames      12   11.0 bytes avg      132 bytes total
G frames     915   11.5 bytes avg    10523 bytes total
E frames       2    9.5 bytes avg       19 bytes total
Frames     41321   30.1 bytes avg  1241961 bytes total
Data rate  285Hz   8659 bytes/s      86600 baud

1 frames failed to decode, rendering 4 loop iterations unreadable. 6 iterations are missing in total (21ms, 0.01%)

kaefert@mint ~/Downloads/quadcopter_stuff/blackbox-log/2015-08-09_1110 try luxfloat PID $ ~/Downloads/quadcopter_stuff/blackbox-log/blackbox-tools/obj/black
box_render LOG00138.TXT 
4355 frames to be rendered at 30 FPS [2:26]

Rendered 500 frames (11.5%)...
Rendered 1000 frames (23.0%)...
Rendered 1500 frames (34.4%)...
Rendered 2000 frames (45.9%)...
Rendered 2500 frames (57.4%)...
Rendered 3000 frames (68.9%)...
Rendered 3500 frames (80.4%)...
Rendered 4000 frames (91.8%)...
Rendered 4355 frames (100.0%).

Units of angle in converted csv file

While converting *.BFL file to *.csv in the headers there are units of barometer (cm), rotational velocity in ex. (deg/s) and acceleration (m/s/s) if the units were selected by command line switches. But there in no command line switch to select units of attitude provided when --simulate-imu is activated.

The orientation angles are not raw values,, are in (deg). To have consistent file, the units of angle should be added to roll, pitch and yaw in headers

It is now:
accSmooth[0] (m/s/s) accSmooth[1] (m/s/s) accSmooth[2] (m/s/s) roll pitch heading

It should be:
accSmooth[0] (m/s/s) accSmooth[1] (m/s/s) accSmooth[2] (m/s/s) roll(deg) pitch(deg) heading(deg)

Generic event logging

@thenickdude : What do you think about adding event header to blackbox protocol? It should be possible to reuse code for I-Frame, maybe without predictor

H Event 20 ID:Gtune result
H Event 20 name:axis,gyroP,newP
H Event 20 signed:0,1,1
H Event 20 predictor:0,0,0
H Event 20 encoding:0,0,0

And either add new encoding or use VB for log events ...

add current PID configuration to blackbox data/render

would be very useful to have the current PID settings as a renderable item for the overlay. will be very helpful for those that use in flight adjustments to tune.

Converting CSV data to actual readings

Hi all, I am trying convert the decoded blackbox CSV data into actual conventional sensor readings that we are viewing on Blackbox explorer for my own analysis. Would like to ask if there are any guidelines on how I can do this?

Thanks in advance!

Feature request: aircraft type

Change icon to different aircraft (at least the most common for inav: wing, plane) to match controls (1/2 motors and 2 to 4 servos)
It would be nice if naming will be changed on graph to, at least from motors to servos, even for motor channel

Wing icon as example with 1 motor and 2 servos. Servos should be not circles, but deflection up/down

Missing field encoding header for field #0 'GPS_fixType'

Hi all, I am running a Lumenier F7 dual gyro f7 controller with the Matek F422SE firmware on INAV. I recorded a flight using the blackbox logger with a Beitian 880 GPS and compass module and tried to view the log. The explorer is unable to parse the data and returns the following error:
Missing field encoding header for field #0 'GPS_fixType'

WIth Blackbox decode tools I was able to get three files out, but they are very short and do not contain the logs.

Use blackbox_decode with a .csv file

The log file produce by my Taranis is already a .csv file but I still want to be able to produce a .gpx file with the tool for displaying gps data on google earth.

But I can't figure how to make it work as the tool requires a .txt file ...

Decode does not work with new betaflight .BFL file format

Feature request: customisable graphs

Select graphs to draw by name. Group it. ((Name, data_name, data_name), (Name, data_name, data_name), (Name, data_name, data_name))

Something like (("Attitude", "atttitude[0], "attitude[1]"), ("RC Input", "RC Command [roll]", "RC Command [pitch]"))
In this case we will have two graphs (Attitude and RC Input). On first one w will have two lines ('atttitude[0]' and 'attitude[1]'), on second one two lines too (RC Command [roll], RC Command [pitch])

With such option we can draw any kind of graph we need for any king of aircraft

not able to run program

When i click on the application, the window pops up but then goes away, not sure what going on here

(Enhancement)

I'm having trouble syncing HD Video to the BlackBox File. Is there any way to simplify this? I wish there was a way to movie the video separately and the BlackBox Log File separately so I can just sync them that way.

correct gpx time by absolute time of blackbox start

In gpx decoding the file only contains the relative timestamp from the beginning of logging. When there was a GPS Fix on Arm there is an absolute timestamp in the log in the ASCII Part (At least with inav)

Use that timestamp to correct gpx timestamps to be able to use the gpx to correlate with camera images with "gpxcorrelate".

Header of blackbox log file:

[...]
H Field S encoding:1,1,7,7,7,1
H Firmware type:Cleanflight
H Firmware revision:INAV 1.9.1 (03a5c1922) OMNIBUSF4PRO_LEDSTRIPM5
H Firmware date:Apr 21 2018 13:15:03
H Log start datetime:2018-06-16T13:10:49.264+00:00
H Craft name:
H P interval:1/1
[ ... ]

Only relative timestamps in gpx file:

<trk><name>Blackbox flight log</name><trkseg>
  <trkpt lat="52.3008762" lon="8.5960688"><ele>57</ele><time>2000-01-01T00:02:02.389080Z</time>

feature request: Log start timestamp to output filename (logged by iNav 1.8+)

I have attempted here:
jelinj8@f2e005d
but that just prints the date to stderr.

suggestions:

• Naming output files
• fixed GPX output timestamps
• optional timestamp column in output (e.g. for DashWare telemetry rendering)

My C++ is too rusty for that.

Header lines look like this:
H Firmware revision:INAV 1.9.1 (03a5c1922) MATEKF405
H Firmware date:Apr 21 2018 13:25:29
H Log start datetime:2018-05-04T18:17:08.376+00:00

Trouble is that the filenames (and files) are created before parsing the log. Probably renaming (replacing a placeholder) the files before memfreeing the filenames would be the simplest solution?

Poor performance on Rpi3 ubuntu server

Hi,

I'm developing a bot that uses blackbox decode and the performance is really poor under this system. I understand that the HW used is not the best for this task, and using best hw will improve the way it works.

As i see on htop, the software is just using one core to perform the actions, I don't know if it will be possible to do something to improve runtime:

here is a capture of the htop command showing the cpu usage:

And the task itself:

as you can see, PID task 10052 is been running now for almost half an hour, when on a windows 10 system (it takes about 1 or 2 minutes top).

Is anything I can do? I've downloaded and compiled the code on the raspberry pi 3. Log file:

This is de log that i'm using to test it:

btfl_004.zip

Thanks.

blackbox-render: More options

A few more options would be nice:

• Hide "Batt. cell" (bottom left corner)
• Hide "Acceleration" (bottom left corner)
• Change colors

Raw data to rad/s to deg/s

Hi,

I'm wondering how the raw data from the gyro's is converted to rad/s and then deg/s

Increase max number of Logs in file

I use 'blackbox_decode' for retrieving GPS tracks from iNav logs.
Sometimes it happens that a single file contains more than 31 logs.
For this reason I increased constant FLIGHT_LOG_MAX_LOGS_IN_FILE.
Is there any reason why is it limited now? Could it be increased in next release?

blackbox-tools/src/parser.h

Line 10 in 3c0b9ea

#define FLIGHT_LOG_MAX_LOGS_IN_FILE 31

Thank you!

Feature Request: Output Tuning Values to Text File from blackbox_decode

When decoding log files from blackbox_decode it would be great if the PID tuning values used during that log were output to a text file (rather than having to keep a manual log for optimising tuning). I think they're already decoded as part of blackbox_render but that makes it difficult to batch-analyse files

Feature request: acceleration output in Earth coordinate system

In blackbox_decode the acceleration out in done in controller coordinate system. It would be useful to have possibility to output accelerations in the Earth coordinate system, with gravity removed (az-1g).
It is done in what way in blackbox_render, so it should be quite simply.
All functions already exists in the code blackbox_render.
It is just a matter of output these values to .csv file by blackbox_decode.

Proposed command line switch for blackbox_decode:
--acceleration-csys acceleration output coordinate system (controller|Earth), default is controller

compute PID error and RC rate

blackbox_decode does not compute RC rate and PID error (for anyone interested in doing offline analyses on these). My current workaround is to load the log in explorer and save an csv file from there which contains these variables, so presumably they are computed in explorer. I am trying to do offline Matlab analyses on PID error, and it'd be less cumbersome, and easier to implement within Matlab scripts, if blackbox_decode did this.

Deadlock while reading modern betaflight 3.2.2 file

Hi,

I have just upgraded my setup to betaflight 3.2.2, and now the blackbox-tools all seem to deadlock on my blackbox logs.
These logs were created on the SD card in an SPRacing F3 NEO board, used to work nicely with cleanflight in the past.

Even with the raw and debug-parameters the software just deadlocks and uses 100% CPU.
blackbox_render, blackbox_decode and the Chromium-webapp all show this behaviour.

https://screenshot.tbspace.de/xawdoshiczy.png
https://tbspace.de/content/downloads/LOG00008.BFL

Any ideas on what could be wrong with the new files? (i have 8 of these here, happens with all of them).

Thanks,
Tobias

Bus error: 10

i got this error on 2 different logs, the decode had no errors.
other index on the same log worked.
why do i get it and how i prevent it in the future?

(i use a mac-mini with osx 10.10.4 and the tools 0.4.1)

Dashware

Nick,
Is there any tools available to convert the current blackbox log to something Dashware can read? I couldn't get the old tools to open.

This is from the FAQ on dashware.net
I want to make my own data file. How can I make it work with DashWare?
Here are some notes on making a DashWare-friendly CSV data file:

-It’s good to have a static line of text at the top declaring that it is your data format. Something like “Some Cool Logger Data File”. The DashWare Data Profile for your format can then be easily set up to automatically detect one of your data files.
-We require a CSV header line: A line of text containing the column headers/names.
-Then the raw data lines would follow after that.
-There should be a Time column (full timestamps work great, e.g. “2011/08/07 20:10:02.200”, or the total running time in seconds e.g. “1.1”)
-Latitude and Longitude should be in +/- decimal degrees
-Be consistent with your data units, or at least change the column name when you use different units. DashWare can be used to create calculated data columns for other units.

You will then need to make a Data Profile that understands your format. Bring up the Data Profile Editor (File Menu > Edit Data Profiles…) and press F1 for detailed help. Tips:
-The column mappings (bottom right area of dialog) are used to map your column names to known DashWare data types. Set up the mapping for your time column first, so you can use the “Test File / View Data…” button. It is very handy. “ / Data Running Time, Seconds” is the mapping you need.
-To get a map to show up, you need to use our “Degrees to Cartesian Converter” (calculators/converters are in the bottom left area of dialog) to create X and Y units in feet. You must map these new X and Y output columns to our “Positional \ Longitude / X Position (Feet)” and “Positional \ Latitude / Y Position (Feet)” data types.

Have a look at an existing data profile, such as the GPX one, as an example.

New feature, 3 Axis G meter

It would be neat to have a 3 axis G meter. A peak hold indicator would be nice also. The most I have hit was 2.8 G's with KK2.0, F330. Thanks guys, BB is great!

Add support for decoding current sensor data

Add support for decoding current sensor data in blackbox_render and blackbox_decode