textually-duplicate passages in msmarco v2 about msmarco

[craswell]

Yeah, we knew dupes will be an issue with the new datasets. It's a realistic dataset right now, but in such realistic situations we know there would be deduping mechanisms in the retrieval system, to prevent users from seeing duplicate stuff.

If people participate in the passage task by first ranking documents, then ranking the passages of the top-k documents, then we definitely don't want to remove any passages, just because they also appeared in some other doc. So if we wanted to do some unrealistic form of deduping, it might look more like having a single passage ID that points to many different documents, so no passage-document connections are lost.

Having said that, as soon as we get rid of exact dupes then there are "near dupes" to consider.

Overall our approach so far was to do some testing to make sure the collection is usable with the current training+dev sets, which seems to be the case, and we'll have to figure some further steps later. Thanks for raising the issue.