incineratorr / gan-speech-synthesis

GAN-Speech-Synthesis

The model described in the provided code is a Generative Adversarial Network for Text-to-Speech (TTS) synthesis, referred to as GAN-TTS. It consists of a feed-forward generator and two discriminators. Here's a summary of what this model can do:

Text-to-Speech (TTS) Synthesis: The main purpose of this model is to generate speech (mel-spectrograms) from input text. Given a text input, the generator produces corresponding mel-spectrograms that represent the speech.

Speech Realism Discrimination: The model has two discriminators - discriminator_realism and discriminator_utterance. The discriminator_realism evaluates the realism of the generated mel-spectrograms (speech) to distinguish between real and fake audio. This is a binary classification task.

Text Embedding Discrimination: The discriminator_utterance evaluates the embeddings of the input text to distinguish between real and fake text embeddings. This is another binary classification task.

Training for Better Speech Generation: The generator and discriminators are trained in an adversarial manner to improve the quality of generated speech. The generator aims to generate realistic mel-spectrograms to fool the discriminators, while the discriminators try to accurately classify real and fake mel-spectrograms and text embeddings.

Data Preprocessing: The code includes functions for preprocessing audio files (mel-spectrograms) and text data (text embeddings). Mel-spectrograms are extracted from audio, and text embeddings are obtained using a pre-trained BERT model.

TPU and GPU Compatibility: The code supports training on both TPUs and GPUs. If a TPU is available, the training will be done on TPU, otherwise on GPU or CPU.

Logging and Visualization: The model can log training metrics and visualize results using Weights and Biases (WandB) platform. This allows easy monitoring of the training progress and results.

Overall, the GAN-TTS model in this code aims to generate high-quality speech from input text, leveraging adversarial training to improve the realism of the generated speech.