Flexible creating of parameters in Training? about mamba

[Hprairie]

In Mamba B and C are functions of x (the first part of the xz variable). To create a B and C, I'm guessing they use x_proj to get something like this B, C, delta = x_proj(x), which means that B, C, and delta are input dependent. If you want them to be static, i.e. not time-variant, then you can set them to parameters like in the S4 papers. However, the paper abalates that selectivity is important for language so it would be best to leave them as functions of x if the downstream task in language.

That being said, their cuda kernel does technically allow B and C to be static and not inputs of x, but again this has been shown to have worse downstream performance.

[tridao]

By default, B and C are the results of x_proj(x) (i.e. B and C are input-dependent), which is done inside mamba_inner_fn. If you pass sth not None to mamba_inner_fn, those would be used instead. It might work, but that code path is used extensively and as a result not as well-tested.