Different behaviours on VisDA-2017 using different pretrained models from timm about sdat

[swift-n-brutal]

After some research, the first problem has been resolved. The default behaviour of ViT.forward() changed in different version of timm. When global_pool='', the backbone returns x[:, 0] in timm=v0.5.x, while it returns x in timm=v0.6.7.

[rangwani-harsh]

Hi @swift-n-brutal are you now able to get the correct accuracy for VisDA dataset?

[swift-n-brutal]

I can get a close result (89.8%) of CDAN+MCC+SDAT on VisDA, but met a strange behaviour. As shown in the image below, the validation accuracy (not the mAP) keeps going down as the training proceeds, and the best result (mAP 89.9%) is achieved only at the first epoch. Then I wondered whether the pretrained model was problematic. I tested two models: vit_g ('https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.01-res_224.npz') from timm=0.5.x and vit_jx ('https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_p16_224-80ecf9dd.pth') from timm=0.4.9. For vit_g the acc goes down, while for vit_jx the acc increases but the final mAP is (88.6%) much lower than the former one.

[Wangzs0228]

I can get a close result (89.8%) of CDAN+MCC+SDAT on VisDA, but met a strange behaviour. As shown in the image below, the validation accuracy (not the mAP) keeps going down as the training proceeds, and the best result (mAP 89.9%) is achieved only at the first epoch. Then I wondered whether the pretrained model was problematic. I tested two models: vit_g ('https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.01-res_224.npz') from timm=0.5.x and vit_jx ('https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_p16_224-80ecf9dd.pth') from timm=0.4.9. For vit_g the acc goes down, while for vit_jx the acc increases but the final mAP is (88.6%) much lower than the former one.

Have you found the reason? Is it a flaw in the model or a problem with our operation?

[swift-n-brutal]

@Wangzs0228 It is almost sure that the smoothness regularization is beneficial to transferability, robustness, generalization ability, etc. For a specific task, the results may vary. I am not working on this task recently. You can try the experiments to see if the results match your expectations.