Leveraging Attack Non-Transferability to Boost Adversarial Robustness for Foundation Models
Source abstract
Abstract
This paper presents a novel adversarial defense framework that strategically exploits the non-transferability of adversarial attacks across multi-modal foundation models. While Contrastive Language–Image Pre-training (CLIP) models demonstrate remarkable zero-shot capabilities, they remain vulnerable to adversarial samples. Adversarial fine-tuning is widely adopted as a standard defense, yet the resulting robustness against sophisticated white-box attacks is often insufficient. To address this limitation, we aim to boost the robustness of an adversarially fine-tuned model by utilizing a pre-trained auxiliary model to leverage attack non-transferability. Specifically, we construct a common embedding space and introduce a detection scheme that identifies the attack target based on feature distances. By adaptively switching the prediction output, we effectively mitigate attacks. Experimental results demonstrate that our approach outperforms state-of-the-art adversarial fine-tuning methods in terms of adversarial robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan
Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan
Funding
JSPS KAKENHI
References (34)
Show source-provided references
1. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, E., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv.
2. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (March, January 26). Scaling up visual and vision-language representation learning with noisy text supervision. Proceedings of the International Conference on Machine Learning, Shenzhen, China.
3. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (March, January 26). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Shenzhen, China.
4. Kato, N., Nota, Y., and Aoki, Y. (2024). Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification. Sensors, 24.10.3390/s24113624 ↗
5. Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., and Li, Y. (2022, January 19–24). RegionCLIP: Region-based language-image pretraining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01629 ↗
6. Asperti, A., Naldi, L., and Fiorilla, S. (2025). An Investigation of the Domain Gap in CLIP-Based Person Re-Identification. Sensors, 25.10.3390/s25020363 ↗
7. Aljuhani, H., Dahab, M.Y., and Alsenani, Y. (2025). Enhancing Document Classification Through Multimodal Image-Text Classification: Insights from Fine-Tuned CLIP and Multimodal Deep Fusion. Sensors, 25.10.3390/s25247596 ↗
8. Elhenawy, M., Ashqar, H.I., Rakotonirainy, A., Alhadidi, T.I., Jaber, A., and Tami, M.A. (2025). Vision-language models for autonomous driving: Clip-based dynamic scene understanding. Electronics, 14.10.3390/electronics14071282 ↗
9. Nguyen, N., Vu, M.N., Ta, T.D., Huang, B., Vo, T., Le, N., and Nguyen, A. (2025, January 19–23). Robotic-CLIP: Fine-tuning CLIP on action data for robotic applications. Proceedings of the IEEE International Conference on Robotics and Automation, Atlanta, GA, USA.10.1109/ICRA55743.2025.11127829 ↗
10. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (May, January 30). Towards deep learning models resistant to adversarial attacks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
11. Croce, F., and Hein, M. (2020, January 13–18). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. Proceedings of the International Conference on Machine Learning, Online.
12. Mao, C., Geng, S., Yang, J., Wang, X., and Vondrick, C. (2023, January 1–5). Understanding zero-shot adversarial robustness for large-scale models. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.
13. Wang, S., Zhang, J., Yuan, Z., and Shan, S. (2024, January 16–22). Pre-trained model guided fine-tuning for zero-shot adversarial robustness. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02313 ↗
14. Schlarmann, C., Singh, N.D., Croce, F., and Hein, M. (2024). Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv.
15. Toishi, K., Maeda, K., Togo, R., Ogawa, T., and Haseyama, M. (2025, January 6–11). Robust adversarial defense based on non-transferability of attack across foundation models. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Hyderabad, India.10.1109/ICASSP49660.2025.10888691 ↗
16. LAION-5B: An open large-scale dataset for training next generation image-text models
17. DataComp: In search of the next generation of multimodal datasets
18. Vishniakov, K., Shen, Z., and Liu, Z. (2024, January 21–27). ConvNet vs transformer, supervised vs. CLIP: Beyond ImageNet accuracy. Proceedings of the International Conference on Machine Learning, Vienna, Austria.
19. Carlini, N., and Wagner, D. (2017, January 22–26). Towards evaluating the robustness of neural networks. Proceedings of the IEEE Symposium on Security and Privacy, San Jose, CA, USA.10.1109/SP.2017.49 ↗
20. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, J. (2021, January 4). An image is worth 16 × 16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
21. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
22. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
23. Goodfellow, L.J., Shlens, J., and Szegedy, C. (2015, January 7–9). Explaining and harnessing adversarial examples. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
24. Tong, B., Lai, H., Pan, Y., and Yin, J. (2025, January 11–15). On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR52734.2025.01855 ↗
25. Xing, S., Zhao, Z., and Sebe, N. (2025, January 11–15). CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR52734.2025.01413 ↗
26. Sheng, L., Liang, J., Wang, Z., and He, R. (2025, January 11–15). R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR52734.2025.02788 ↗
27. Deng, J. (2009, January 20–25). A large-scale hierarchical image database. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
28. Defense strategies for adversarial machine learning: A survey10.1016/j.cosrev.2023.100573 ↗
29. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images. [Master’s Thesis, Department of Computer Science, University of Toronto].
30. Coates, A., Ng, A., and Lee, H. (2011, January 11–13). An analysis of single-layer networks in unsupervised feature learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA.
31. Parkhi, O.M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012, January 16–21). Cats and dogs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248092 ↗
32. Nilsback, M.-E., and Zisserman, A. (2008, January 16–19). Automated flower classification over a large number of classes. Proceedings of the Sixth Indian Conference on Computer Vision, Graphics and Image Processing, Bhubaneswar, India.10.1109/ICVGIP.2008.47 ↗
33. Bossard, L., Guillaumin, M., and Van Gool, L. (2014, January 6–12). Food-101-mining discriminative components with random forests. Proceedings of the IEEE European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10599-4_29 ↗
34. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. (2014, January 23–28). Describing textures in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.461 ↗