juliagusak / dataloaders

Pytorch and TensorFlow data loaders for several audio datasets

dataloaders

Pytorch and TFRecords data loaders for several audio datasets

Datasets

1. ESC - dataset of environmental sounds

• ESC Downloader
• Pytorch DataSet
• TFRecords Loader

2. LibriSpeech - corpus of read English speech

• LibriSpeech downloader for PyTorch
• PyTorch DataSet
• PyTorch DataSet for TFRecord
• PyTorch DataLoaders for TFRecord
• TFRecords Loader
• TFRecords Generator

3. NSynth - dataset of annotated musical notes

• NSynth downloader and generator of *.h5py and *.tfrecord formats
• TFRecord reader
• PyTorch Dataset
• PyTorch Dataset for TFrecord
• PyTorch DataLoaders for TFRecord

4. VoxCeleb2 - human speech, extracted from YouTube interview videos

• Pytorch loader
• TFRecords loader

5. GTZAN - audio tracks from a variety of sources annotated with genre class

• GTZAN Downloader
• PyTorch DataSet

6. CallCenter - audio tracks with human and non-human speech

• PyTorch DataSet

For validation we frequently use the following scheme:

1. Read 10 random crops from a file;
2. Predict a class for each crop;
3. Averaging results.

For this scheme we've done additional DataLoaders for PyTorch:

• DataLoader for ESC, GTZAN, LibriSpeech
• DataLoader for LibriSpeech from TfRecords
• DataLoaders for NSynth

dataloaders's Issues

about hdf5

I have several hundred GB wav files on my disk (about 1, 000 hours wav data). I found directly reading the wav file is slow for training, so I choose lmdb and hdf5 as options. However I found that
hdf5 do not support concurrent, i.e. num_workers in Dataloader can not be more than 1, how do you solve this problem? thx