Can I train my custom MLP model with DEQ in tutorial page? about deq

[jerrybai1995]

Hi @san9569 ,

Yes, ReLU should be fine! In fact, you may want to check this paper: https://arxiv.org/pdf/2006.08591.pdf. They also use ReLU (but parameterized W differently to ensure provable convergence).

When you use your proposed MLP as layer f, it may get unstable. In that case, you can (1) ensure your parameters are initialized to very small values, (2) add weight decay if needed, and (3) add normalizations if needed.

[jerrybai1995]

I'm closing this issue for now, but do let me know (and feel free to re-open) if you have other questions!

[eugenebak]

I'm closing this issue for now, but do let me know (and feel free to re-open) if you have other questions!

Thanks for your kind answer and for recommending the paper!