axxxjt / fluenteditor

[InterSpeech'2024] FluentEditor:Text-based Speech Editing by Considering Acoustic and Prosody Consistency

FluentEditor: text-based speech editing by considering acoustic and prosody consistency

This repo contains official PyTorch implementations of:

• Fluenteditor: text-based speech editing by considering acoustic and prosody consistency (Interspeech 2024) Demo page | Code

This repo contains unofficial PyTorch implementations of:

• FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models (ACL 2023) Demo page

• CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing (ICASSP 2022)
  Demo page

• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (ICML 2022)
  Demo page | Official code

• EditSpeech: A text based speech editing system using partial inference and bidirectional fusion (ASRU 2021)
  Demo page

Supported Datasets

Our framework supports the following datasets:

• VCTK

Downloading VCTK

You can download the VCTK dataset from the official website. Follow these steps:

1. Visit the VCTK dataset download page.
2. Download the dataset (VCTK-Corpus.tar.gz).

Extract the downloaded file to your desired directory. For example:

tar -xzf VCTK-Corpus.tar.gz -C ../data

Install Dependencies

Please install the latest numpy, torch and tensorboard first. Then run the following commands:

export PYTHONPATH=.
# install requirements.
pip install -U pip
pip install -r requirements.txt
sudo apt install -y sox libsox-fmt-mp3

Finally, install Montreal Forced Aligner following the link below:

https://montreal-forced-aligner.readthedocs.io/en/latest/

Download the pre-trained vocoder

mkdir pretrained
mkdir pretrained/hifigan_hifitts

download model_ckpt_steps_2168000.ckpt, config.yaml, from https://drive.google.com/drive/folders/1n_0tROauyiAYGUDbmoQ__eqyT_G4RvjN?usp=sharing to pretrained/hifigan_hifitts

Data Preprocess

# The default dataset is ``vctk``.
python data_gen/tts/base_preprocess.py
python data_gen/tts/run_mfa_train_align.sh
python data_gen/tts/base_binarizer.py

Train

# Example run for Fluenteditor.
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --dir /path/to/your/fluenteditor --config egs/fluenteditor.yaml --exp_name fluenteditor --reset

Inference

We provide the data structure of inference in inference/example.csv. text and edited_text refer to the original text and target text. region refers to the word idx range (start from 1 ) that you want to edit. edited_region refers to the word idx range of the edited_text.

id	item_name	text	edited_text	wav_fn_orig	edited_region	region
0	1	"I'd love to be at the world cup."	"I'd absolutely love to be at the world cup."	inference/audio_example/1.wav	[1,3]	[1,2]

# run with one example
python inference/tts/fluenteditor.py --exp_name fluenteditor

Evaluation

# Example Objective Evaluation for Fluenteditor.
# You can use the following objective evaluation metrics: MCD, STOI, PESQ
python eval/get_metrics.py

License and Agreement

Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.

Tips

1. If you find the mfa_dict.txt, mfa_model.zip, phone_set.json, or word_set.json are missing in inference, you need to run the preprocess script in our repo to get them. You can also download all of these files you need for inferencing the pre-trained model from https://drive.google.com/drive/folders/1BOFQ0j2j6nsPqfUlG8ot9I-xvNGmwgPK?usp=sharing and put them in data/processed/vctk.
2. Please specify the MFA version as 2.0.0rc3.

If you find any other problems, please contact me.