light1726 / speechtriplenet

The implementation of paper "SpeechTripleNet: End-to-End Disentangled Speech Representation Learning for Content, Timbre and Prosody"

SpeechTripleNet

The implementation of paper "SpeechTripleNet: End-to-End Disentangled Speech Representation Learning for Content, Timbre and Prosody"

Demo | Paper | Checkpoint

Environment setup

conda env create -f environment.yml
conda activate speechtriplenet-env

Quick try of speech editing with pretrained model

# Download the pretrained model from https://drive.google.com/file/d/1dAdPXtENtACtVokBZyzn32DlWd1Zk3Yy/view?usp=sharing;
# Put it under output-CCDPJ-c_100.0_1.3-s_10.0_60.0-p_10.0_3.0/ckpt/VCTK/
jupyter notebook speech_editing.ipynb

Data feature extraction for training

python preprocess.py --config ./configs/VCTK/preprocess.yaml

Training

CUDA_VISIBLE_DEVICES=0 python train.py --mdl CCDPJ -p ./configs/VCTK/preprocess.yaml -t ./configs/VCTK/train.yaml -m ./configs/VCTK/model.yaml

Inference

See speech_editing.ipynb.

speechtriplenet's Issues

Regarding the shape of the generated Mel-Spectrograms

Hello. After preprocessing the dataset, I am getting shape of the mel-spectrograms as (80,1). It has 80 mel bands which is fine but how can it only have one 1 number of frames? With the model you have for the content posterior etc you need an input of B,C,T. B and C are sorted but with only one dimension at T its not working.

How to generate Test & Train path file named "test-prosody.txt" and "train-prosody.txt"?

I tried to run the demo speech_editting file and there are some issue about these dataset path files, I wonder the format of these files. Thanks a lot