Dataset metadata for audio_codec about nemo

[rlangman]

If you are interested, I opened a PR yesterday (#8723) which contains a tutorial notebook you could download and try https://github.com/NVIDIA/NeMo/blob/79ffef97d2993b716406227f00fc234c19541610/tutorials/tts/Audio_Codec_Training.ipynb. Change the branch from 'main' to 'codec_tutorial'.

The training manifest only needs {"audio_filepath", "duration"}

The metadata structure is defined at https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/data/vocoder_dataset.py#L45. For each dataset you use it asks for the manifest_path to the manifest file, and the root audio_dir where the audio is stored. If training on multiple datasets you can specify sample_weight to determine how often it samples from each.

train_ds_meta configures training data, val_ds_meta configures data for computing validation statistics/loss, and log_ds_meta is used with this callback (https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/parts/utils/callbacks.py#L331) to configure logging of various artifacts during training (eg. reconstructed audio, embedding plots). If doing local logging then log_dir specifies the local directory.

A full training command (copied from the tutorial), with 1 dataset "vctk", looks like:

python /content/nemo/examples/tts/audio_codec.py
--config-path=/content/nemo/examples/tts/conf/audio_codec
--config-name=audio_codec_24000.yaml
max_epochs=10
weighted_sampling_steps_per_epoch=10
batch_size=4
log_dir=/content/exps/EnCodec/test_run/logs
exp_manager.exp_dir=/content/exps
+exp_manager.version=test_run
model.log_config.log_wandb=False
model.log_config.log_tensorboard=False
model.log_config.generators.0.log_dequantized=True
trainer.accelerator=gpu
+train_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/train_manifest.json
+train_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
+val_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
+val_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
+log_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
+log_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed

Let me know if you have any other questions.

[kthworks]

Thank you for your kind reply. It was so helpful to me !!