Request for the best model weight for CATH 4.2 testset about proteinmpnn

[sokrypton]

Have you tried temperature of zero?

[EricZhangSCUT]

Have you tried temperature of zero?

I tried and got an error of div by zero

[dauparas]

Hello,
Thank you! The publicly shared models are the ones trained on protein assemblies in the PDB (as of Aug 02, 2021) since those models have been used for lab experiments and design rescues described in the paper. We used CATH 4.2 only to show how small changes in the input features and architecture can lead to large improvements compared with the baseline model described before (Ingraham et al.). The test accuracy numbers shown in the Table 1 (ProteinMPNN paper) are calculated by taking argmax over amino acids given a native sequence autoregressive context, so there is no temperature or sampling involved in this evaluation. It makes sense that you are getting sequence recovery a couple percent lower when sampling with low temperature compared with using argmax with native sequence autoregressive context. I would guess that your setup is correct.

[EricZhangSCUT]

Hello, Thank you! The publicly shared models are the ones trained on protein assemblies in the PDB (as of Aug 02, 2021) since those models have been used for lab experiments and design rescues described in the paper. We used CATH 4.2 only to show how small changes in the input features and architecture can lead to large improvements compared with the baseline model described before (Ingraham et al.). The test accuracy numbers shown in the Table 1 (ProteinMPNN paper) are calculated by taking argmax over amino acids given a native sequence autoregressive context, so there is no temperature or sampling involved in this evaluation. It makes sense that you are getting sequence recovery a couple percent lower when sampling with low temperature compared with using argmax with native sequence autoregressive context. I would guess that your setup is correct.

Thank you for replying. I am wondering if it is possible for you to share the best model weight on CATH 4.2 testset?
Since I tried to train the 50.8-recovery model in Table 1 Experiment 4 with prorein_mpnn_utils.py and the setting in ProteinMPNN paper (v_30_000, transformer lr schedule, batch size 6000, dropout 0.1, random decoding) while the mean and median recovery on CATH 4.2 testset is respectively 42.01 and 44.35. It would be very helpful if you make the model weight avaliable.

[EricZhangSCUT]

Sorry. I just realized that 'given a native sequence autoregressive context' means it worked as the forward function with known sequences but not sampling with only structures. So actually I should not compare the results with sampled sequence recovery from my experiment or other works. Right?

[dauparas]

Correct, you could compare perplexities (Table 1, PDB Test Perplexity) as it is done here: https://github.com/jingraham/neurips19-graph-protein-design/blob/master/experiments/train_s2s.py#L183

[EricZhangSCUT]

Thank you! It helps a lot.