houwenxin / zerospeech2020

Code for ZeroSpeech 2020

ZeroSpeech 2020

Implementation of our submission to ZeroSpeech 2020 (Hou et al.)

Results: https://zerospeech.com/2020/results.html

The system is composed of a hierarchical VQ-VAE encoder to discover discrete spoken word units from speech and a MelGAN vocoder to directly generate speech. They are trained separately. During VQ-VAE training, we add the speaker id before the VQ-VAE decoder to help reduce the speaker information encoded in the word units.

Usage:

1. Run scripts/data_manifest.sh (modify datadir to the root dir of raw dataset)
2. python trainer.py --mode vqvae --language [english/surprise] --ckpt_path [path-to-save-model] --datadir [path-to-the-root-dir-of-dataset]
3. python trainer.py --mode melgan --load_vqvae True --pretrained_vqvae [path-to-vqvae-ckpts] --language [english/surprise] --ckpt_path [path-to-save-model] --datadir [path-to-the-root-dir-of-dataset]
4. python evaluator.py --language [english/surprise] --datadir [path-to-the-root-dir-of-dataset] --vqvae_model [path-to-vqvae-ckpts] --melgan_model [path-to-melgan-ckpts] --save_path [path-to-save-generated-results]

Referred Repositories:

1. vq-vae-2-pytorch: https://github.com/rosinality/vq-vae-2-pytorch
2. melgan-neurips: https://github.com/descriptinc/melgan-neurips
3. ZeroSpeech-TTS-without-T: https://github.com/andi611/ZeroSpeech-TTS-without-T
4. VQ-VAE-Speech: https://github.com/swasun/VQ-VAE-Speech
5. pytorch-vqvae: https://github.com/houwenxin/pytorch-vqvae

zerospeech2020's Issues

train melgan

Thanks for sharing your code. I have a question when running your code.
When training the melgan which is used to synthesis speech, there is a parameter --load_vqvae [path-to-vqvae-ckpts], how should I set it? I try to give the path to pretained model and also try 'True', but both are wrong.
trainer.py: error: unrecognized arguments: ./model1/model-vqvae-5-148-744-True.pt
trainer.py: error: unrecognized arguments: True

ABX

Hi.
I'm trying to change the model based on your code. But when I run the ZS20 evaluate code to get the ABX score, I found that after hundreds of epochs training for the melgan, the ABX stays the same. Do you have this problem in your work?

melgan

Hi, I have a problem when I train the melgan, the best model will always be the first epoch's model. Then I check the log and find it seems can't convergence.

Lr of melgan is 2e-4 which is the same as your hps, how to solve this problem?