Can not reproduce this issue about diffcse

[Gitsamshi]

Same here, got the result of 77.33.

[hbin0701]

Me too.

[qmin2]

Here also :(
Below is my result, with batch size and etc. same as written in paper.

[voidism]

Hi all!

After I tried some experiments on another machine, I found that the hyperparams are very sensitive to the device you use. I cannot reproduce the results on another machine with the same hyperparams either. Your python/pytorch/cuda/huggingface version will affect your results. The hyperparams on the paper are only suitable for the first machine I used, so you probably need to re-search hyperparams to get the same results on your machine.

A recent CSE paper https://github.com/yiren-jian/NonLing-CSE/tree/main/VisualCSE seems also have this problem for CSE. They suggest use the same hardwares and software versions to faithfully reproduce the results in the paper.