keonlee9420 / diffsinger Goto Github PK

Your implementation has diffusion_projection for every residual block similar to DiffWave, but this is inconsistent with the paper as the original architecture directly adds E_t (output of the step embedding module) to the input before the first convolution layer. Is there a reason behind this change?

hi, thanks for your work. but i met a problem in diffsinger and diffgan both
Traceback (most recent call last):
File "F:\DiffSinger-main\preprocess.py", line 19, in
preprocessor.build_from_path()
File "F:\DiffSinger-main\preprocessor\preprocessor.py", line 113, in build_from_path
if len(f0) > 0:
UnboundLocalError: local variable 'f0' referenced before assignment
it bothers me a lot, thanks for your help

https://diffsinger.github.io/ returns a 404. It worked ~1 year ago (as I recall) but not now.

Hi Guys.

thank you very much for your great work!

I have one question regarding the function of get_energy_embedding in modules.py.

I see during training if the target enegy values is not none, the model uses the target ones to generate enegy embeddings instead of the predicted ones? Why?

def get_energy_embedding(self, x, target, mask, control):
        x.detach() + self.predictor_grad * (x - x.detach())
        prediction = self.energy_predictor(x, squeeze=True)
        if target is not None:
            embedding = self.energy_embedding(torch.bucketize(target, self.energy_bins))
        else:
            prediction = prediction * control
            embedding = self.energy_embedding(
                torch.bucketize(prediction, self.energy_bins)
            )
        return prediction, embedding

Conversely, the model uses the predicted pitch to generate pitch embeddings.

def get_pitch_embedding(self, decoder_inp, f0, uv, mel2ph, control, encoder_out=None):
        pitch_pred = f0_denorm = cwt = f0_mean = f0_std = None
        if self.pitch_type == "ph":
            pitch_pred_inp = encoder_out.detach() + self.predictor_grad * (encoder_out - encoder_out.detach())
            pitch_padding = encoder_out.sum().abs() == 0
            pitch_pred = self.pitch_predictor(pitch_pred_inp) * control
            if f0 is None:
                f0 = pitch_pred[:, :, 0]
            f0_denorm = denorm_f0(f0, None, self.preprocess_config["preprocessing"]["pitch"], pitch_padding=pitch_padding)
            pitch = f0_to_coarse(f0_denorm)  # start from 0 [B, T_txt]
            pitch = F.pad(pitch, [1, 0])
            pitch = torch.gather(pitch, 1, mel2ph)  # [B, T_mel]
            pitch_embed = self.pitch_embed(pitch) 

Could you please help to answer it?

Thank you!

Does SVS work in English's lyrics?

Hi, thanks for your excellent implementation. I would like to know how to apply this code to a multi-speaker setup, such as LibriTTS? More specifically, how can I add speaker embedding into the pipeline? Your response would help me a lot. Thanks in advance!!

In this case, , i ran the scripts python3 train.py -p config/vietnam/preprocess.yaml -m config/vietnam/model.yaml -t config/vietnam/train.yaml
File "train.py", line 199, in
main(args, configs)
File "train.py", line 85, in main
losses = Loss(batch, output)
File "/home/thanhdo/envs/diffsinger_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/thanhdo/Documents/DiffSinger/model/loss.py", line 69, in forward
log_duration_targets = log_duration_targets.masked_select(src_masks)
RuntimeError: The size of tensor a (39) must match the size of tensor b (136) at non-singleton dimension 1

have you try using sing data？