The error samples are due to issues with the ground truth annotations rather than errors in the model predictions. about mixvpr

[amaralibey]

Hello @libenchong,

Thank you again for your valuable feedback. Yes, I've noticed some errors in that regard but never did a thorough investigation like you did. That would make a great study case I think.

Best regards,

[libenchong]

Hello@amaralibey. In these error cases, the images predicted by the model to be most similar to the query are not in the ground truth, or the geographical distance between the predicted images and the query is slightly larger than the ground truth. I'm not sure how to handle these cases, do you have any good suggestions?
this is error sample.I think the images predicted by the model are more accurate than the ground truth

[amaralibey]

Hello @libenchong,

Great observation!

In most benchmarks, the ground truth (GT) typically includes images located within a 25-meter perimeter from the query, based on GPS coordinates. However, since Mapillary images are primarily crowdsourced from various sources like phones and dashboard cameras, mapillary_sls often exhibit a considerable amount of noise, as demonstrated in the example you provided.

Although the model's prediction appears to be close to the query (judging by the door on the right), the noisy coordinates associated with the image indicate a distance greater than 25 meters, resulting in an inaccurate label.

I would propose manual verification when creating a benchmark, specifically for positive matches within the 25-40 meter range from the query, particularly when we are relying solely GPS coordinates that are prone to a lot of noise. This way, we can ensure a more precise evaluation of the model's performance.

Additionally, it's worth noting that these errors in the labels apply to all evaluated techniques and might affect their performance in a similar manner. This helps maintain fairness in the evaluation process.

Again, thank you for your valuable observations.