0913ktg / vits_korean_multispeaker

Korean multi-speaker VITS

This project was implemented using the official PyTorch implementation by "jaywalnut310". After training for 10 epochs (32 batch size, 460k steps), the inference results for two random male and female speakers are available in the inference_samples.

Getting Started

Setting up the development environment for this project can be challenging due to version conflicts with various libraries.

Therefore, we managed the development environment of this project using a Docker container.

The Docker image, which can be used to create the Docker container, can be downloaded from Docker Hub.

Dataset

The data used for model training can be downloaded from the following link.

• 문학작품 낭송․낭독 음성 데이터(시, 소설, 희곡, 시나리오)
• 감성 및 발화스타일 동시 고려 음성합성 데이터
• 감성 및 발화 스타일별 음성합성 데이터

Data Preprocessing

데이터 세트를 다운로드 한 뒤 학습할 수 있게 전처리를 해야 합니다.

1. If file paths contain Korean or special characters, change them to English or numbers.
2. Convert .wav files for training to a 22kHz sampling rate.
3. If there are stereo files, convert them to mono.
4. The filelists downloaded from Google Drive link the labels and wav files. (.cleaned files are the result of converting sentences using g2pk.) 4-1. If using a different phoneme conversion module, convert a few English words to Korean pronunciation and remove the '\xa0' special character.
5. Place the make_mels.py file at the top path of the dataset and generate melspectrograms. (Pre-create files required for training.)

Installing

You can clone this GitHub repository and use it.

git clone https://github.com/0913ktg/vits_korean_multispeaker

You can download the model checkpoints and filelists from the Google Drive link.

Train

Once you have 22kHz audio files, train, and validation filelists, and have completed data preprocessing, you can start training by running train_ms.py. Multi-GPU usage has been confirmed.

Synthesis

1. In inference.py, modify the path for the Generator checkpoint accordingly.
2. Enter the desired Korean sentences in texts. (Separate multiple sentences with a comma.)
3. Enter the speaker number in sid. (from 0 to 184)
4. Running inference.py will create a file named test{i}.wav.

vits_korean_multispeaker's Issues

please let me torch version and pip package version

Please let me know the touch version.
What is the cuda version?
What is the package version?

Bert-VITS2 Korean version

Hello DaegyeomKim,

I am also trying to training a Korean version of Bert-VITS2. Have you training the Korean models successfully?

Thank you for sharing your excellent code. Have a great day.

Best regards.

22kHz sampling rate 변환 문의

안녕하십니까, 음성 합성과 관련해 연구하고 있는 김민규입니다.

데이터 전처리 시 .wav 파일들의 samplling rate를 22kHz로 변환하라고 하셨는데
AI Hub의 데이터들의 sampling rate를 어떻게 변환하셨는지 알려주실 수 있으실까요?

감사합니다.

sid 별 화자 정보 질문드립니다.

안녕하세요, 인공지능 연구하고 있는 김민규입니다.

Pre-trained 모델에서 sid가 185개로 설정이 가능한데 학습 데이터를 봤을 때,
구연체, 중계체 등 다양한 대화 스타일 및 기쁨, 슬픔 등 감정들이 들어 가 있는 것 같습니다.

혹시 각 index가 어떤 스타일 또는 감정을 가지고 있는지 구분할 수 있는 정보가 있을까요?
있으시다면 알려주시면 감사하겠습니다.