ellington-project / ellington

Automated tempo estimation for swing dance DJs

Ellington

Ellington is an experimental project to automate the calculation of beats-per-minute (BPM) information for swing jazz music, such as the works of Duke Ellington - the project's namesake. BPM information for swing jazz music is notoriously hard to calculate automatically, as the shuffle of the rhythm section, the soft/loud chunking guitar, and overall swing feel mean that standard algorithms (which are often optimised for four-on-the-floor feel music) report inaccurate times. As such, this project has two main goals:

1. Provide a platform for experimenting with various BPM algorithms and tools (machine learning anyone?) in order to find high quality (at least >90% accuracy, 99% of the time) solutions.

2. Provide a tool, or set of tools, for automatically processing libraries of swing jazz music, and reporting BPM information.

30-Second overview

Ellington is based around the idea of a 'library' of audio files, that you might wish to process. This is similar to an iTunes library - you must explicitly add audio files to the library, and where possible metadata is written to the library, not the audio file.

An example ellington usage flow might be:

ellington init library.json -d ~/Music/
ellington bpm library.json 
ellington write library.json

The above commands, in order:

• Initialise a new ellington library library.json
• Calculate bpm information for each audio file in the library, storing the information in library.json
• Write the bpm information to the audio file comment - where requested (more on this later).

NOTE: ELLINGTON IS PRE-ALPHA, AND VERY BUGGY. BACK UP YOUR MUSIC LIBRARY BEFORE USING IT TO WRITE METADATA TO AUDIO FILES!!

Dependencies

Most Ellington dependencies are expressed using the rust package manager, cargo, and so will be automatically installed when Ellington is built. However, Ellington makes use of a number of external programs for tasks such as parsing mp3 audio data, or writing id3v2 tags. These are:

• ffmpeg
• id3v2
• mp4info
• mp4tags

External programs are listed in src/shelltools, in case any are missing here.

Detailed Usage

Ellington currently supports four different operations (see ellington --help for more information):

• Library initialisation: ellington init
• BPM calculation: ellington bpm
• Writing ellington metadata to audio files: ellington write
• Clearing ellington metadata from audio files: ellington clear

Library initialisation

Ellington library files can be initialised as follows:

ellington init <library_file> -d <directory> 
ellington init <library_file> -i <itunes_xml_library> 
ellington init <library_file> 

This will write a json-based library to the file given in <library_file. Audio discovery is currently possible with three different methodologies. Ellington can:

• Recursively explore a directory tree to find music files -- -d <directory>
• Read individual audio file names from stdin -- -i <itunes_xml_library>
• Read an iTunes XML library to discover music files (but not other metadata) -- no further parameters.

Bpm calculation

Ellington can, given a library file, calculate the bpm of each track in the library using a "pipeline". A pipeline is a combination of an audio decoder (e.g ffmpeg), and a bpm algorithm (e.g. naive). The results of the bpm calculations are written in place to the library file. This stage can be invoked as follows:

ellington bpm <library_file> 

Metadata writing

Ellington files, themselves, are not that useful for the casual DJ. It takes a while to find each song in the JSON, and JSON itself can be a bit tricky to read. In order to remedy this, Ellington can write the data that it has calculated to the audio file itself, as follows:

ellington write <library_file> 

NOTE: This will modify metadata of the audio files listed in library_file. Run this command at your own risk - it may damage your audio library!

As the bpms calculated with Ellington are not yet high quality, Ellington avoids writing to an audio file's bpm tag, but instead writes a specially formed piece of text to the comment field of the audio data. Ellington is (by default) very polite - it only writes to the comment when requested.

Comments with Ellington metadata contain a valid Ellington data string of the form:

[ed#<data>#de]

Where <data> is a JSON string, with : replaced with #, representing some Ellington data (see src/library/ellingtondata.rs).

A good default Ellington data string is:

[ed#{"algs"#{}}#de]

In order to persuade Ellington to write to an audio file, edit the 'comment' metadata tag of it to include the above data string, using your tag editor of choice. Alternatively, Ellington can be made more aggressive, by passing the --append flag to the write command. This will append the ellington data to an existing comment even if it does not yet contain comment data.

Debugging

By default, Ellington is quite conservative in what it prints. In order to get it to log more, export the following environment variable as follows:

RUST_LOG=ellington,libellington

Feature Targets

0.1.0: (current master)

• Audio file discovery through iTunes based libraries
• Audio file discovery through recursive directory enumeration
• Audio file discovery through stdin
• Support for generic audio decoding using ffmpeg
• Support for mp3 tagging using id3v2
• Support for mp4 metadata parsing using mp4info
• Support for mp4 metadata writing using mp4tags
• Naive BPM calculation algorithm acting on raw audio data
• Draft json-based ellington-data format for ephemeral bpm information
• Comment appending (i.e. programmatically marking tracks as wanting to have bpm information written to them)

0.1.1:

• Updated/better ellington-data metadata format for audio data tags.

0.1.2:

• Support for FLAC metadata

0.1.3:

• Integration tests for tag reading/writing of mp3,mp4,FLAC files

0.1.4:

• Unit tests for library components

0.2.0:

• Replace id3v2 program invocation with library calls.
• Replace mp4tags and mp4info program invocations with library calls.

0.3.0:

• Stream output/input for libraries (i.e. writing a library to stdout, reading one from stdin - this should allow us to pipe libraries between ellington commands)

0.4.0:

• Integration of static ffmpeg libraries instead of system calls
• Integration of all dependencies in cargo.toml
• Standalone binary, without dynamic dependencies (including external programs)

0.5.0

• Parallel bpm analysis

1.0.0:

• Stable release of Ellington.
• Windows support

2.0.0:

• Neural network based bpm classifier

ellington's Issues

Ensure that mp3 files have id3v2 tags before writing

At present, ellington writes comments (if --append is passed) to mp3 audio files without id3v2 data, which takes precedence over id3v1. Unfortunately, some applications then ignore all id3v1 data - essentially breaking the track.

Add id3tag support for writing results to comments.

We need to be able to mark the "calculated" bpms somewhere that we can see it, but not rely on it. The id3tag comment section seems to be a good place for this...

Fix build & linker errors on travis

Shared libraries need to be packaged with ellington on all platforms

Currently (to avoid dll-hell) we package libtag with ellington on windows. We need to do the same on OSX and linux as well.

Add option to run estimators when creating library/cache

At the moment the library/cache creation command is not able to run estimators on the tracks that have been discovered. It would be good to add this behaviour, and potentially clean up the CLI implementation somewhat.

Better error handling

As of right now, there are 26 unwrap calls in the codebase. Any of those could panic, and should be handled properly!

Add support for windows equivalents of id3v2 and mp4tags

The command line programs id3v2 and mp4tags are not available on windows. In order to support windows, Ellington either needs to use a platform agnostic library for managing metadata or find equivalent windows programs.

mp4 tags: https://code.google.com/archive/p/mp4v2/ (or equivalent)
id3v2 tags: any...

Why is a library required?

It seems to me that (unless you have other goals) the primary usefulness of this tool would be as something used by a plugin to another audio player. In that context, I imagine that ellington maintaining it's own library would just get in the way and make it more difficult to integrate with an external audio player.

From that perspective, it seems that it would make sense to separate the processing pieces from the library pieces (note: haven't actually looked through the code yet) and have a separate executable that takes in an audio file and spits out some JSON to stdout with BPM data. This would make it trivial to integrate with an external audio player.

Background: I'm the primary maintainer of the Exaile audio player, which I use to DJ swing music. Currently I have an autodetect bpm plugin which uses the SoundTouch library (as a gstreamer plugin). I'd be willing to integrate ellington as an option if it really works as well as you say.

Parallelise analysis stage

With the current naive bpm tools algorithm each track takes roughly a second to process. Scaling this up, this coul dmean a long processing time for a large library. Ideally, we would be able to parallelise this process.

Remove dependence on bpm-tools

We don't want to rely on the C implemented bpm-tools. Although it's easy to call, and use, ideally we'll have it in Rust at some point, which should allow us to tweak it and make it faster as we need to.

Replace explicit ffmpeg call with link of ffmpeg library

e.g. integrate it with a C interface, such as: https://rodic.fr/blog/libavcodec-tutorial-decode-audio-file/

Add a license

Can a license be added for this project? I'm assuming since it's been shared publicly for people to try out it's open-source and free for anyone to use and modify but if that isn't your intent that would be good to know :)

Replace python call to bellson with static linked library

Bellson should be packaged as a rust library that we can import, rather than hackily calling python over the command line.

One-shot tool for integration with external audio players

As a developer of an audio player, I'd like to integrate Ellington into my application to do BPM autodetection.

The best implementation would be a single static binary with no external dependencies that could be executed by the audio player. However, a nonstatic binary would be ok if the dependencies were manageable.

Some requirements I can think of for a player/platform agnostic interface with an executable tool:

• Input audio (any one of the below options seems ok)
  • Path to an audio file via command line, or
  • Audio file contents via stdin, or
  • Raw decoded audio via stdin
• Output would be on stdout as JSON
• Ideally, no library would be required -- but if it were, then having the tool do that automatically would be best
  • One solution could be creating a library in a temporary directory, and deleting the library after bpm detection is finished

One thing that an external audio player definitely would not want is ellington modifying the audio file that was passed in. Any audio player is going to have its own way of managing audio file tags, and that should be left up to the application. Additionally, I imagine that the audio player would want to prompt the user whether the autodetected value is acceptable.