zhuchen03 / gradinit Goto Github PK

Hello,

Thanks for such a great work. Auto-initializing a DNN in a proper way definetely sounds amazing.

Yet, the usability needs to be significantly improved so that I can plug this in my existing networks.

It would be great if that could be as easy as installing and then importing an additional package.

We should maybe open a feature request in PyTorch so that they integrate this into the framework.

Hello! Thank you for a great paper. Could you please explain the idea behind the division of the loss value by eta? I didn't see any information related to this in the paper. In my experiments, I haven't observed any significant difference between the code with/without this division, but maybe I am missing something important.

I would love to test your method out on language modeling tasks in fairseq.

Do you have the code to make table 2 (or just the GradInit rows in Table 2) handy?