Examples of speech tensor shape for gst models? about espnet_model_zoo

[kan-bayashi]

I added multi-speaker example in the notebook.
Please enjoy!
https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_tts_realtime_demo.ipynb

[kan-bayashi]

Hi. @ajbouh.
Since the feature extraction will be performed on-the-fly, you must provide wavform for speech (e.g. torch.randn(50000,).

[ajbouh]

Thank you for the example! As I experiment with it, I see that the length of the supplied speech is very important. I also see that the output seems to be slower and the quality varies quite widely.

Here's a trimmed down example for two different reference speakers and some text (both reference and random): https://colab.research.google.com/drive/1FLcSi4fSqkxfwzfVRMwzPQgpGp-FOXpJ?usp=sharing

Do you have any advice for improving quality?

[kan-bayashi]

Hmm. I did not tune the hyperparameters so much for GST, especially the following parameters.

• gst_heads: the number of attention heads.
• gst_tokens: the number of GST tokens.
  https://github.com/espnet/espnet/blob/5aeb9926334125d63121e7a11764aa8ac21dee67/egs2/vctk/tts1/conf/tuning/train_gst_tacotron2.yaml#L34-L35

We need to tune these parameters to make it stable to reflect the speaker's characteristics.
If you can share the experimental results with us, that is great :)

[ajbouh]

What sorts of experimental results do you mean?

Do you want the samples I generated from the current model checkpoint or are you suggesting something else?

[kan-bayashi]

The behavior in the synthesis is also important.
And we need to tune the above hyper parameters.
So if you train the model with different parameters and report the effectiveness, that is great.

[ajbouh]

Ah, I see. Unfortunately I don't have a single GPU that's as good as what you've been using. How long are your training runs on the V100? Do any of your scripts support multi-GPU training?

…

On Sat, Sep 19, 2020, 21:21 Tomoki Hayashi ***@***.***> wrote: The behavior in the synthesis is also important. And we need to tune the above hyper parameters. So if you train the model with different parameters and report the effectiveness, that is great. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABZHMSKBJ7MBQKNK3PUTTSGV7LJANCNFSM4RRCEKLQ> .

[kan-bayashi]

GST Tacotron2 takes 2 days on Titan V.
Of course, we support multi-gpu training.

[ajbouh]

I have a machine I can use for long term training with multiple 1080Ti cards. Hopefully that will be enough!

Which part of the documentation should I look at to see how to retrain GST Tacotron2 on the same dataset you've been working with?

[kan-bayashi]

Try https://github.com/espnet/espnet/tree/master/egs2/vctk/tts1
See the usage https://github.com/espnet/espnet/tree/master/egs2/TEMPLATE/tts1