Some issues about implementing DurationPitchPredictor about naturalspeech2-pytorch

[JiaYK]

@lucidrains Thank you very much for your reply!
Your efficiency is really high, you released a new version so quickly.
Regarding the implementation of the third point, I see that you use ResnetBlock, each ResnetBlock contains 2 convolutions and SiLU (it seems to be generally better than Relu), does this mean that each ResnetBlock is equal to 2 convolutions? I'm not very sure~ Maybe it really needs running experiments to determine the effect

[JiaYK]

There is another small issue. When reproducing the NS2 paper, I used Torchinfo to count the parameter count. Each module follows the settings written in the paper (default configuration for those not written) and will never match.

For example, in the Phoneme Encoder, I changed the FeedForward in the code to the Convolutional FeedForward mentioned in the paper. When I set the Conv1D Filter Size of two 1D convolutions to 2048 and Kernel Size to 9, the parameter quantity is 119M. Only when one Kernel Size is 9 and the other Kernel Size is 1 can the difference be not much, which is 69M. It's not equivalent to 72M in the paper

Or Audio Codec, I initialized it as described in the paper

from audiolm_pytorch import SoundStream, EncodecWrapper
SoundStream(rq_num_quantizers=16, 
codebook_size=1024, 
codebook_dim=256,
strides=(2, 4, 5, 5))

the parameter quantity is 35M, only by modifying channels and use_local_attn can only be changed to the 27M mentioned in the paper. How did you solve the problem of mismatched parameter quantities? Or ignore it? Is it acceptable if the order of magnitude is correct?

Looking forward to your recovery very much!

[lucidrains]

@JiaYK yup, just the general ballpark is ok

from my point of view, the field is an approximate science rather than delicate engineering. hit on a few main ideas and it should be fine

[lucidrains]

@JiaYK hey yes, i am aware of 1 and will get that done today!

for 2, RMSNorm has been used successfully in a number of large language models by now (alphacode, llama). I think we can safely start using it

for 3, i was actually not sure about that, but will be adding residuals (it will be redone as resnet blocks)

[lucidrains]

@JiaYK ok, 1 should be done, will probably get 3 done today too

[lucidrains]

@JiaYK ok, 3 should be done too! let me know what you think

[lucidrains]

@JiaYK oh true, yea, let's make that a hyperparameter

[lucidrains]

@JiaYK although i guess it is still sensitive to certain decision in the neural network. not very apt description

let's just say... very different than traditional engineering

[JiaYK]

Thank you very much for your reply~ Then I will not worry about the specific parameters, and give priority to realizing all the functions!