Global query: wrong repeating about gcvit

[pamolchanov]

We indeed refactored our internal version of the code and introduced some bugs trying to simplify things. Thanks @tridao @shkarupa-alex for helping to identify those issues. We have fixed them and retraining models to share in the repo. ETA is EOW.
It is worth to mention that current release weights still replicate numbers in the paper. We expect new models to do better.

[ahatamiz]

@shkarupa-alex Thanks for your comment. I tested it again but get the correct order of images in the batch for both train and validation. We will test again using different Pytorch versions and hardware to ensure consistency.

[shkarupa-alex]

@ahatamiz you can reproduce issue in such way:

1. Add "import numpy as np" at the top of https://github.com/NVlabs/GCVit/blob/main/models/gc_vit.py
2. Add following code after this line https://github.com/NVlabs/GCVit/blob/main/validate.py#L224

            batch_size = input.shape[0]
            batch_range = np.arange(batch_size, dtype='int32')
            np.random.shuffle(batch_range)
            input = input[batch_range]
            target = target[batch_range]

3. Run evaluation with any batch size > 1 multiple times

You will always get different top1 and top5 accuracy

[tridao]

I ran into the same issue when playing with the current GCViT model code. It does seem that repeating q_global is mixing examples from different batches.
Perhaps there's an internal version of the model code that's different from this public version?

[ahatamiz]

Hi @tridao

We have updated our models and re-trained them. In all cases, the accuracy is better compared to reported numbers in the paper, from our internal checkpoints. In addition, we have conducted a set of stress tests and verified there is no inconsistency with the model performance across different batch sizes at test time. Please check the new model weights and let us know if you face any issues.