Visualizing Attention about protein_bert

[nadavbra]

@miladansari I'm not sure what BertViz is, but keep in mind that ProteinBERT uses global attention which is a 3D tensor (batch x heads x seq_len) as opposed to most transformers that use self-attention which is a 4D tensor (batch x heads x seq_len x seq_len). This is actually part of the point of using global attention - it is more compact and easier to visualize. On our notebook we have a section with code for visualizing ProteinBERT's attention. I hope that helps.

[miladansari]

Great, Thanks