No test split for Mind Small about mteb

[nreimers]

@Muennighoff:
As many people use MSMARCO for training, maybe it make sense to keep it as a dev dataset?

[nreimers]

@NouamaneTazi Yes, renaming dev to test is fine.

[NouamaneTazi]

cc @nreimers, Is it okay to rename the dev split as test for tasks that don't have test splits?
source link for MIND

[Muennighoff]

We could also leave it at dev I guess, we just need to clarify it in the benchmarking. It would avoid confusion with the actual hidden test set of MindSmall used for the leaderboard here.

Similarly, for MSMARCO we will have to run the dev set anyways I think, since it's the one BEIR used.

[NouamaneTazi]

I may be wrong, but the leaderboard shows scores for the original MIND dataset, while this dataset is MIND-small, and I'm not sure it's intended to have a test set?

And for MSMARCO, yes I agree as long as we keep the same split names

[nreimers]

For simplicity I would just rename the dataset from dev to test, instead of having a configuration that tells that for MIND you should use the 'dev' split for the test leaderboard.

In the paper we can then just mention that we use the dev split for MIND as test set for MTEB

[Muennighoff]

@NouamaneTazi yes you're right, my bad!
@nreimers Makes sense, agreed! For MSMARCO we'll have to tell people to run dev though