Official code implementation of our work:

Chengze Jiang, Minjing Dong, Xinli Shi, Jie Gui. Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference. To appear in AAAI 26 Oral.

Abstract: Vision-language pre-training models (VLPs) demonstrate strong multimodal understanding and zero-shot generalization, yet remain vulnerable to adversarial examples, raising concerns about their reliability. Recent work, Test-Time Counterattack (TTC), improves robustness by generating perturbations that maximize the embedding deviation of adversarial inputs using PGD, pushing them away from their adversarial representations. However, due to the fundamental difference in optimization objectives between adversarial attacks and counterattacks, generating counterattacks solely based on gradients with respect to the adversarial input confines the search to a narrow space. As a result, the counterattacks could overfit limited adversarial patterns and lack the diversity to fully neutralize a broad range of perturbations. In this work, we argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense. Accordingly, we propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating orthogonal gradient directions and momentum-based updates. This design expands the exploration of the counterattack space and increases the diversity of perturbations, which facilitates the discovery of more generalizable counterattacks and ultimately improves the ability to neutralize adversarial perturbations. Meanwhile, we present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength. Extensive experiments on 16 datasets demonstrate that DOC improves adversarial robustness under various attacks while maintaining competitive clean accuracy.

Ensure that Conda is installed, and set up the environment using the dependencies listed in requirements.txt.

Please download and unzip all the raw datasets into ./data. It's okay to skip this step because torchvision.datasets will automatically download (most of) them as you run the code if you don't already have them.

The main implementation is provided in Code/DOC.py.

To reproduce the results reported in Table 1 of the manuscript, execute the following command:

python Code/DOC.py \
    --batch_size 256 \
    --test_attack_type 'pgd' \
    --test_eps 4 \
    --test_numsteps 10 \
    --test_stepsize 1 \
    --outdir 'DOC_results/Github' \
    --seed 1 \
    --DOC_eps 4 \
    --beta 2 \
    --DOC_numsteps 4 \
    --DOC_stepsize 3.0 \
    --learnable_tau 0.155 \
    --temperature 75.0 

To reproduce the results reported in Table 2 of the manuscript, execute the following command:

python Code/DOC.py \
    --batch_size 256 \
    --test_attack_type 'CW' \
    --test_eps 4 \
    --test_numsteps 10 \
    --test_stepsize 1 \
    --outdir 'DOC_results/Github' \
    --seed 1 \
    --DOC_eps 4 \
    --beta 2 \
    --DOC_numsteps 3 \
    --DOC_stepsize 3.0 \
    --learnable_tau 0.155 \
    --temperature 75.0 

Our code is developed based on the open-source code of TTC (CVPR-25). We thank the authors for their brilliant work. Please also consider citing their paper:

@InProceedings{Xing_2025_CVPR,
    author    = {Xing, Songlong and Zhao, Zhengyu and Sebe, Nicu},
    title     = {CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {15172-15182}
}