hcy71o / sc-cnn

SC-CNN: Effective Speaker Conditioning Method for Zero-Shot Multi-Speaker Text-to-Speech Systems

SC-CNN : Effective Speaker Conditioning Method for Zero-Shot Multi-Speaker Text-to-Speech Systems

Thanks to StyleSpeech and VITS, we built up our codes based on Link and Link

0. VCTK dataset is used.
1. Sampling rate is set to 22050Hz.
2. This is the implementation of SC-TransferTTS

Materials

• Demo page
• Paper

Prerequisites

0. Clone this repository.
1. Install python requirements. Please refer requirements.txt
   1. You may need to install espeak first: apt-get install espeak
2. Download datasets
   1. Download and extract the VCTK dataset, and downsample wav files to 22050 Hz. Then rename or create a link to the dataset folder: ln -s /path/to/VCTK-Corpus/downsampled_wavs DUMMY3
3. Build Monotonic Alignment Search and run preprocessing if you use your own datasets.

# Cython-version Monotonoic Alignment Search
cd monotonic_align
python setup.py build_ext --inplace

Training Exmaple

python train.py -c configs/vctk_base.json -m vctk_base

Inference Example

See inference.ipynb

sc-cnn's Issues

About additional loss

Hello, nice work. I have a question.
Q) how about adding an extra loss at the end of generation to match the spk_enc of reference wav and generated wav? Because I do not see meta-stylespeech's discriminator being used here? (am i missing it somewhere?)

....
s_ref = self.spk_enc(y.transpose(1,2), (y_mask==0).squeeze(1))
....
## freeze spk_enc
s_out = self.spk_enc(y_out.transpose(1,2), (y_out_mask==0).squeeze(1))
# then cosine dist b/w s_ref and s_out
## unfreeze spk_enc

Thanks.

A problem about the kl loss

Hi, I tried to run the code, but the kl_loss is always infinity.

Acoustic feature transfering question

Hello, I'm new to this filed and still a little bit confused, I got few questions about the acoustic feature transfer.

1. In my understanding, this zero shot TTS transfer the speaker's voice from reference audio to do the synthesis, I'm wondering will it also transfer the speech style such as pitch, intonation, speaking rate, rhythm, volume, or emotional expression?
2. If this zero shot TTS does not transfer speech style, do you have any suggestion for doing style control or provide any kind of controllability base on VITS model?

I understand these questions are not that relevant to the repo issue, but any kind of help is appreciated, thank you!!

about the paper

Do you have published the paper?