Any suggestions for regularization? about mamba

[chazzmoney]

I’m very interested to see the response here from the authors, but I’ll say something potentially wrong.

i’m not familiar with the architecture used to create MIDI using the transformer mechanism, but in the original MAMBA paper they were pretty clear that continuous signals do not benefit form MAMBA, and that something like S4 seemed to outperform. So the performance you may be seeing may have less to do with regularization and more to do with the application of MAMBA to this problem.

[tridao]

You can use dropout, just like Transformers. It's not implemented here but you can add it.

[ElliottDyson]

You can use dropout, just like Transformers. It's not implemented here but you can add it.

This thread may be more useful if tagged as a feature request then? (Just for traceability to mitigate it from being lost)

[Anri-Lombard]

This is still an issue; I trained the model on 10 epochs for a specific use case and the loss just decreases drastically to overfit quickly

[Anri-Lombard]

Has anyone implemented this with some success?

[windsornguyen]

I find that adding dropout decreases performance for state space models. Does anyone else also observe this phenomenon?

[Anri-Lombard]

I added dropout and it actually improved performance for me.

[Anri-Lombard]

Although MAMBA still does very bad in my use case, it just learned better.