This is a repository for the paper, VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis, ICASSP 2025.
VoiceDiT is a multi-modal generative model for producing environment-aware speech and audio from text and visual prompts.
git clone https://github.com/kaistmm/VoiceDiT.git
cd VoiceDiT
pip install -r requirements.txt- Generate audio with description prompt and content prompt:
python generate.py --desc_prompt "She is talking in a park." --cont_prompt "Good morning! How are you feeling today?"- Generate audio with audio prompt and content prompt:
python generate.py --modality "audio" --desc_prompt "assets/bird_chirping.wav" --cont_prompt "Good morning! How are you feeling today?"- Generate audio with image prompt and content prompt:
python generate.py --modality "image" --desc_prompt "assets/park.jpg" --cont_prompt "Good morning! How are you feeling today?"- Text-to-Speech Example:
python generate.py --desc_prompt "clean speech" --cont_prompt "Good morning! How are you feeling today?" --desc_guidance_scale 1 --cont_guidance_scale 9- Text-to-Audio Example:
python generate.py --desc_prompt "trumpet" --cont_prompt "_" --desc_guidance_scale 9 --cont_guidance_scale 1- Image-to-Audio Example:
python generate.py --desc_prompt "assets/tiger.png" --cont_prompt "_" --v2a_guidance_scale 2 --desc_guidance_scale 9 --cont_guidance_scale 1Generated audios will be saved at the default output folder ./outputs.
View the full list of options with the following command:
python generate.py -hThis work would not have been possible without the following repositories:
