about 'An Easier Trick for Boundary Prediction' about diffsinger

[MoonInTheRiver]

In your paper, we can get the predicted boundary as follows:

then I implemented 'An Easier Trick for Boundary Prediction' in my repo following the trick: https://github.com/keonlee9420/DiffSinger/blob/d5dbe05ee1c7da0878393c73129089a67d0fe935/boundary_predictor.py#L14-L45

and there are some helper functions for that (please focus on expected_kld_t and expected_kld_T function): https://github.com/keonlee9420/DiffSinger/blob/d5dbe05ee1c7da0878393c73129089a67d0fe935/model/diffusion.py#L351-L389

But as I noted in my README.md (in 2. of note section), the predicted boundary of LJSpeech is 100, which is the same as the total timesteps in Naive version.

So I'd like to ask you to briefly check my implementation. Could you please take a look at it and let me know if I missed something? Why do you think my boundary predictor shows unexpected K_step?

FYI, here is the sample output log of running boundary_predictor.py:

==================================== Prediction Configuration ====================================
 ---> Total Batch Size: 48
 ---> Path of ckpt: ./output/ckpt/LJSpeech_shallow_el_4
================================================================================================
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:08<00:00,  1.34it/s]
[tensor(6959.2134, device='cuda:0'), tensor(933.3702, device='cuda:0'), tensor(403.9860, device='cuda:0'), tensor(249.2317, device='cuda:0'), tensor(183.4001, device='cuda:0'), tensor(149.2621, device='cuda:0'), tensor(129.2204, device='cuda:0'), tensor(116.2622, device='cuda:0'), tensor(107.4923, device='cuda:0'), tensor(101.0867, device='cuda:0'), tensor(96.2093, device='cuda:0'), tensor(92.4524, device='cuda:0'), tensor(89.3728, device='cuda:0'), tensor(86.7645, device='cuda:0'), tensor(84.4990, device='cuda:0'), tensor(82.5240, device='cuda:0'), tensor(80.7848, device='cuda:0'), tensor(79.1111, device='cuda:0'), tensor(77.5320, device='cuda:0'), tensor(76.0396, device='cuda:0'), tensor(74.6199, device='cuda:0'), tensor(73.2726, device='cuda:0'), tensor(71.9328, device='cuda:0'), tensor(70.6272, device='cuda:0'), tensor(69.2854, device='cuda:0'), tensor(68.0120, device='cuda:0'), tensor(66.7351, device='cuda:0'), tensor(65.4260, device='cuda:0'), tensor(64.1837, device='cuda:0'), tensor(62.9117, device='cuda:0'), tensor(61.6452, device='cuda:0'), tensor(60.3592, device='cuda:0'), tensor(59.0823, device='cuda:0'), tensor(57.8210, device='cuda:0'), tensor(56.5481, device='cuda:0'), tensor(55.2716, device='cuda:0'), tensor(54.0141, device='cuda:0'), tensor(52.7686, device='cuda:0'), tensor(51.4833, device='cuda:0'), tensor(50.2068, device='cuda:0'), tensor(48.9261, device='cuda:0'), tensor(47.6881, device='cuda:0'), tensor(46.4407, device='cuda:0'), tensor(45.2071, device='cuda:0'), tensor(43.9496, device='cuda:0'), tensor(42.7181, device='cuda:0'), tensor(41.5266, device='cuda:0'), tensor(40.2994, device='cuda:0'), tensor(39.1266, device='cuda:0'), tensor(37.9398, device='cuda:0'), tensor(36.7822, device='cuda:0'), tensor(35.6130, device='cuda:0'), tensor(34.5006, device='cuda:0'), tensor(33.3484, device='cuda:0'), tensor(32.2580, device='cuda:0'), tensor(31.1593, device='cuda:0'), tensor(30.1051, device='cuda:0'), tensor(29.0614, device='cuda:0'), tensor(28.0244, device='cuda:0'), tensor(27.0115, device='cuda:0'), tensor(26.0248, device='cuda:0'), tensor(25.0589, device='cuda:0'), tensor(24.1051, device='cuda:0'), tensor(23.1736, device='cuda:0'), tensor(22.2743, device='cuda:0'), tensor(21.3856, device='cuda:0'), tensor(20.5282, device='cuda:0'), tensor(19.6825, device='cuda:0'), tensor(18.8733, device='cuda:0'), tensor(18.0839, device='cuda:0'), tensor(17.3134, device='cuda:0'), tensor(16.5815, device='cuda:0'), tensor(15.8417, device='cuda:0'), tensor(15.1426, device='cuda:0'), tensor(14.4522, device='cuda:0'), tensor(13.8025, device='cuda:0'), tensor(13.1645, device='cuda:0'), tensor(12.5432, device='cuda:0'), tensor(11.9491, device='cuda:0'), tensor(11.3789, device='cuda:0'), tensor(10.8328, device='cuda:0'), tensor(10.2960, device='cuda:0'), tensor(9.7815, device='cuda:0'), tensor(9.2841, device='cuda:0'), tensor(8.8136, device='cuda:0'), tensor(8.3660, device='cuda:0'), tensor(7.9211, device='cuda:0'), tensor(7.5027, device='cuda:0'), tensor(7.1040, device='cuda:0'), tensor(6.7245, device='cuda:0'), tensor(6.3511, device='cuda:0'), tensor(6.0048, device='cuda:0'), tensor(5.6679, device='cuda:0'), tensor(5.3475, device='cuda:0'), tensor(5.0427, device='cuda:0'), tensor(4.7507, device='cuda:0'), tensor(4.4784, device='cuda:0'), tensor(4.2143, device='cuda:0'), tensor(3.9639, device='cuda:0'), tensor(3.7258, device='cuda:0')]
tensor(0.2382, device='cuda:0')

Predicted Boundary K is 100

Thanks in advance!

Hi, I'm celebrating the Chinese Spring Festival these days. 🤗
When using this method to determine K, you should give ground truth F0 to Fastspeech 2 to produce ~M. In this way, ~M and M will have the same harmonics. I think you can understand the reason.
Please note that one should use gt F0 just to determine K. During inference, of course, use the predicted F0 to produce ~M as usual.

[keonlee9420]

Thanks for the quick reply even on the celebrating day!

Yes I understood what you mean, but actually I double-checked that the predictor consumes teacher-forced mel which is generated by inputting gt F0.

duration_target is not None: True
f0 is not None: True

I used this function in GaussianDiffusionShallow to calculate the right-hand side of the inequality in appendix B above:

def q_mean_variance(self, x_start, t):
    mean = extract(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start
    variance = extract(1. - self.alphas_cumprod, t, x_start.shape)
    log_variance = extract(self.log_one_minus_alphas_cumprod, t, x_start.shape)
    return mean, variance, log_variance

so that the expected KLD of step T (which is 100 in our case) is:

@torch.no_grad()
def expected_kld_T(self, x_start, mask, noise=None):
    t = self.num_timesteps # t = T
    x_start, t, mask = self.kld_input(x_start, t, mask)

    mu, _, logvar = self.q_mean_variance(x_start, t)
    mu, logvar = (mu.squeeze(1) * mask), (logvar.squeeze(1) * mask)
    kld = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    kld = kld / mask.sum()
    return kld

but the results show more than 10 times smaller than every expected KLD of step t (which is <= 100) from the function below (and that's why the predicted value for K is 100), which is the direct implementation of the left-hand side of the inequality in appendix B above:

@torch.no_grad()
def expected_kld_t(self, x_pred, x_gt, t, mask):
    x_pred, t, mask = self.kld_input(x_pred, t, mask)
    x_gt, *_ = self.kld_input(x_gt)

    coef = extract(self.alphas_cumprod / (2 * self.log_one_minus_alphas_cumprod.exp()), t, x_pred.shape)
    kld = F.mse_loss(self.noised_mel(x_pred, t), self.noised_mel(x_gt, t), reduction='none')
    kld = (kld * mask).sum() / mask.sum() # or kld.mean() ?
    kld = coef[0].squeeze() * kld
    return kld

would it be matter for the issue?

[MoonInTheRiver]

Thanks for the quick reply even on the celebrating day!

Yes I understood what you mean, but actually I double-checked that the predictor consumes teacher-forced mel which is generated by inputting gt F0.

duration_target is not None: True
f0 is not None: True

I used this function in GaussianDiffusionShallow to calculate the right-hand side of the inequality in appendix B above:

def q_mean_variance(self, x_start, t):
    mean = extract(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start
    variance = extract(1. - self.alphas_cumprod, t, x_start.shape)
    log_variance = extract(self.log_one_minus_alphas_cumprod, t, x_start.shape)
    return mean, variance, log_variance

so that the expected KLD of step T (which is 100 in our case) is:

@torch.no_grad()
def expected_kld_T(self, x_start, mask, noise=None):
    t = self.num_timesteps # t = T
    x_start, t, mask = self.kld_input(x_start, t, mask)

    mu, _, logvar = self.q_mean_variance(x_start, t)
    mu, logvar = (mu.squeeze(1) * mask), (logvar.squeeze(1) * mask)
    kld = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    kld = kld / mask.sum()
    return kld

but the results show more than 10 times smaller than every expected KLD of step t (which is <= 100) from the function below (and that's why the predicted value for K is 100), which is the direct implementation of the left-hand side of the inequality in appendix B above:

@torch.no_grad()
def expected_kld_t(self, x_pred, x_gt, t, mask):
    x_pred, t, mask = self.kld_input(x_pred, t, mask)
    x_gt, *_ = self.kld_input(x_gt)

    coef = extract(self.alphas_cumprod / (2 * self.log_one_minus_alphas_cumprod.exp()), t, x_pred.shape)
    kld = F.mse_loss(self.noised_mel(x_pred, t), self.noised_mel(x_gt, t), reduction='none')
    kld = (kld * mask).sum() / mask.sum() # or kld.mean() ?
    kld = coef[0].squeeze() * kld
    return kld

would it be matter for the issue?

Have you made sure that ~M is correct?
and
Have you scaled ~M to [-1, 1] before the calculation?

We calculated this k a few months ago, and we may have made some approximations at that time. If your problem hasn't been solved, I'll check it again.

[keonlee9420]

which part do i have to check in addition to the ground-truth F0 for ~M?
I can confirm that ~M is normalized before all calculations I mentioned as follows:

@torch.no_grad()
def kld_input(self, x, t=None, mask=None):
    x = self.norm_spec(x)
    x = x.transpose(1, 2)[:, None, :, :]  # [B, 1, M, T]
    if t is not None:
        t = torch.ones(x.shape[0], device=x.device).long() * (t-1)
    if mask is not None:
        mask = ~mask.unsqueeze(-1).transpose(1, 2)
    return x, t, mask

def norm_spec(self, x):
    return (x - self.spec_min) / (self.spec_max - self.spec_min) * 2 - 1

[hxtuniverse]

which part do i have to check in addition to the ground-truth F0 for ~M? I can confirm that ~M is normalized before all calculations I mentioned as follows:

@torch.no_grad()
def kld_input(self, x, t=None, mask=None):
    x = self.norm_spec(x)
    x = x.transpose(1, 2)[:, None, :, :]  # [B, 1, M, T]
    if t is not None:
        t = torch.ones(x.shape[0], device=x.device).long() * (t-1)
    if mask is not None:
        mask = ~mask.unsqueeze(-1).transpose(1, 2)
    return x, t, mask

def norm_spec(self, x):
    return (x - self.spec_min) / (self.spec_max - self.spec_min) * 2 - 1

Hello, I am also following this problem, have you solved it?