Any suggestions for extracting embeddings for sequences with > 1024 residues? about esm

[joshim5]

Yes, during the training process we cropped sequences >1024 sequences, so taking crops is a sensible choice. We haven't experimented with concatenating or averaging the resulting embeddings, but there are a number of things you could try. For example, assuming the sequence is of length 3072, you could:

• split into 3 chunks of length 1024 and concenate
• split into 2048 chunks (start position = 0, 1, 2, ..., 2048) and average appropriately
• split into chunks of different lengths
• many other possibilities...
  It's possible that these will all perform similarly, but as this is still an open research question, we would be eager to hear what you find.

[tomsercu]

Because I'm referencing this issue in the github discussions, let me add another option to Josh' list:
If you know domain boundaries (or can predict them), that would be a good way to split up the protein sequences. Potentially again with averaging the embeddings over a strided window of consecutive domains.

[aliencaocao]

so is there no need to add BOS and EOS in EACH chunk?