With regards to a perceptual loss about ddsp

[jesseengel]

Hi, thanks for the questions! Better perceptual losses is certainly an area for further investigation.

First off, as you can see from the code:
https://github.com/magenta/ddsp/blob/master/ddsp/losses.py

the standard autoencoder configuration
https://github.com/magenta/ddsp/blob/master/ddsp/training/gin/models/ae.gin#L41
actually does use a log-amplitude scaled spectrogram (dB scale).

The configuration that jointly learns to predict frequency
https://github.com/magenta/ddsp/blob/master/ddsp/training/gin/models/ae_abs.gin
uses mel-frequency scale as input to the encoder, and addtionally uses a pretrained CREPE network to capture higher level audio statistics, similar to "perceptual losses" in the image literature.

All models use A-Frequency weighting for calculation of "loudness_db" which is used as a conditioning feature.

There are certainly more options for losses inspired by psychophysics (including several even in losses.spectral_loss(), but we simply haven't used them yet because they haven't yet been necessary.

Feel free to try them out and if you find one that clearly works better, write it up and contribute it to the library.

[PetrochukM]

@jesseengel Thanks for the response!

You didn't quite answer my second question. Let me ask it in a different way...

With regards to the autoencoder configuration...

ddsp/ddsp/training/gin/models/ae.gin

Line 40 in cd98116

SpectralLoss.mag_weight = 1.0

Is there a benefit to training on an amplitude spectrogram? It looks like it's also used in the loss in addition to the log scaled spectrogram.