Compared to self-attention, does Mamba not need multiple heads per layer? Or positional embeddings? about mamba

[tridao]

1. We just apply the selective SSM to each channel / hidden dimension separately, there's no concept of "head". In this sense it's similar to depthwise convolution. The in projection and output projections mixes the channels.
2. There's no explicit position embedding. The model operates in a recurrent way:
   hidden_state at time t+1 = function of hidden_state at time t and input at time t + 1.
   In this sense it's similar to RNNs (e.g. LSTM). The model does have information on which input token comes first, which input token comes right after, etc.

Re: in-context learning: we actually don't know exactly the mechanism the model is using, all this stuff is new and there are lots to explore. One can hypothesize that as the model processes the prompt, it updates its hidden states to store important information, then use that to generate responses.

Transformer circuits describe 1 way transformers can do ICL. A very interesting question is whether other architectures also do ICL that way, or if there's another path.

[hrbigelow]

Great, thank you Tri for the clarification. Very nice to not need positional embedding or individual heads.

[hrbigelow]

Re: in-context learning: we actually don't know exactly the mechanism the model is using, all this stuff is new and there are lots to explore. One can hypothesize that as the model processes the prompt, it updates its hidden states to store important information, then use that to generate responses.

Transformer circuits describe 1 way transformers can do ICL. A very interesting question is whether other architectures also do ICL that way, or if there's another path.

My sense is in-context learning as a concept is at least aspirationally meant to refer to the ability of the model to complete a pattern A B .... A -> B, where A and B are not just tokens, but short token strings. In this setting, 'perfect generalization' would mean successfully predicting B when both A and B strings were never seen before in the training data. In such a setting one can argue from first principles there is no other way to solve this but the model learns to copy the information B from context to output.

In your paper the induction head task was of course much easier than this - a 16 token alphabet, and you are only trying to recall a single token, none of which are novel during testing. (Although the context as a whole is novel) It would be interesting to hit Mamba with the more challenging induction head task.