Error whilst evaluating fine-tuned model with categorical GO terms about protein_bert

[nadavbra]

Hi @mah51,

Just to clarify - you want to fine-tune ProteinBERT to predict the GO annotations which you provide as dataset[1]? How do you encode these GO annotations? What is your output_spec?

[mah51]

Thank you for the response @nadavbra.

Yes I provide the GO annotations as dataset[1], they aren't encoded. The OutputSpec is generated as follows:

output_spec = OutputSpec(OutputType(True, 'categorical'), unique_labels).

I was unaware I had to encode the GO annotations as the model seems to fine-tune with them as a raw input and I assumed they were encoded as part of the encode_dataset method within evaluate_by_len. Thanks.

[nadavbra]

The model was indeed pretrained to predict GO annotations, so it already outputs GO annotation predictions (however, as stated in another issue, there is a small problem with figuring out which output corresponds to which GO annotation). In your case, you may want to fine-tune the model anyway (or pretrain it from scratch), so it can predict the specific GO annotations that you care about (which might be different from the GO annotations we used during pretraining).

You can also pretrain a similar model from scratch with your specific sequences and GO annotations. We have instructions for how to pretrain a new ProteinBERT model.

Anyway, what you are trying to do won't work. You specify a sequential-categorical output, which is not the case for your data (there isn't a different output for each protein residue). You can think about your GO annotations as a set of independent binary predictions (one for each GO annotation).
There isn't a ready-to-use output type for that, meaning you will have to write some code if you want to fine-tune the existing ProteinBERT model. Basically, you will likely want to add a Dense(k, activation = 'sigmoid') layer on top of ProteinBERT's global output, where k is the number of GO annotations you want to predict.

Hope that helps.

[mah51]

Thank you very much for your assistance, I will give that a shot.