GLUs misleadingly named MLP about transformers

[amyeroberts]

@emergenz @younesbelkada @ArthurZucker Outside of the debate of whether it's confusing or not, we're somewhat limited here on renaming. It's possible to rename the class itself (it's not importable from the __init__ and this shouldn't affect existing copies, other than creating a divergence). However, we can't rename the layer attribute in the other layers i.e. self.mlp, as this would change the state dict. As this would then create a mismatch i.e. self.mlp = LlamaGLU(...) I think we should leave as is (possibly with a clarifying comment somewhere in the code).

[emergenz]

Oh, unfortunate but I understand. Thanks for clarification!

[younesbelkada]

Hi @emergenz
Technically you are right, those classes correspond to Gated Linear Units. We could rename them but we need to be careful with public classes as these standalone classes are used by many codebases, e.g.: here to be renamed in order to not affect these - this might introduce some complexity in the modeling codebase so I will let the core maintainers decide on this @amyeroberts @ArthurZucker

[ArthurZucker]

I don't find it misleading at all TBH

[emergenz]

I don't find it misleading at all TBH

@ArthurZucker Can you elaborate? GLUs are not MLPs. In fact, I was looking at the modeling_qwen2.py code when I saw hidden_states = self.mlp(hidden_states) in the DecoderLayer forward pass. I was extremely confused as to why the Qwen models are using MLPs instead of GLUs and therefore checked their technical report, where I realized that they indeed do not use MLPs. Only then did I realize that the Qwen2MLP class indeed represents a GLU.

I would argue that this is highly misleading naming.

[younesbelkada]

Agreed with the statement above !