julianferry / deep-music

Music recognition and generation using deep learning

Deep Music

Objective

Music recognition and generation using deep learning.

Project structure / design:

The project currently has three components, which exist as standalone packages:

1. Preprocessing - convert NSynth audio .wav files to spectrograms with audiolib.
2. Training - train a Convolutional Neural Network to classify instrument spectrograms with PyTorch.
3. Serving - serve the classifier as a RESTful API with flask and gunicorn.

Each package:

• Is callable from the command-line and has configurable parameters. For example, preprocessing is called with:

python -m preprocessing.task
   --data_dir path/to/import/raw/data \
   --job_dir path/to/export/processed/data \
   --filters_dir path/to/import/instrument/filters \
   --config $config \
   --instruments '["keyboard_acoustic", "guitar_acoustic"]'

• Contains a JSON file of run configurations for reproducibility. For example, this preprocessing config file:

  • Gets parsed as $config in the above preprocessing example.
  • Gets exported by the training stage, so that the data used for training can be reproduced.

• Contains shell scripts to run the package locally, with docker, and to deploy the docker image to cloud with a specific configuration ID. Example training scripts

Roadmap

1. Instrument recognition (current):

   • Instrument classification from single note audio
   • Instrument detection from multiple note audio (songs)

2. Genre recognition:

   • Genre classification from songs

3. Music generation:

   • Instrument note generation
   • Musical piece generation
   • Song generation

deep-music's Issues

Move preprocessing to Apache Beam / Dataflow

Preprocessing currently converts all of the data in batch. However, this has a few configuration options and will probably need to be run multiple times with various parameters. It may be better to convert the preprocessing package to process only a single file or a subset of files, based on an input argument.

This would integrate well with Apache Beam, which would allow:

• Running preprocessing only on the required files
• Parallelisation (core feature)
• Leverage multiple CPUs (gcloud Dataflow)
• Simplified tracking of preprocessing configs