Improving Adversarial Robustness via Distillation-Based Purification
Source abstract
Abstract
Despite the impressive performance of deep neural networks on many different vision tasks, they have been known to be vulnerable to intentionally added noise to input images. To combat these adversarial examples (AEs), improving the adversarial robustness of models has emerged as an important research topic, and research has been conducted in various directions including adversarial training, image denoising, and adversarial purification. Among them, this paper focuses on adversarial purification, which is a kind of pre-processing that removes noise before AEs enter a classification model. The advantage of adversarial purification is that it can improve robustness without affecting the model’s nature, while another defense techniques like adversarial training suffer from a decrease in model accuracy. Our proposed purification framework utilizes a Convolutional Autoencoder as a base model to capture the features of images and their spatial structure.We further aim to improve the adversarial robustness of our purification model by distilling the knowledge from teacher models. To this end, we train two Convolutional Autoencoders (teachers), one with adversarial training and the other with normal training. Then, through ensemble knowledge distillation, we transfer the ability of denoising and restoring of original images to the student model (purification model). Our extensive experiments confirm that our student model achieves high purification performance(i.e., how accurately a pre-trained classification model classifies purified images). The ablation study confirms the positive effect of our idea of ensemble knowledge distillation from two teachers on performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Integrated Circuits and Semiconductor Failure Analysis · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Hanyang University, Seoul 04763, Republic of Korea
Division of Nanotechnology, Daegu Gyeongbuk Institute of Science & Technology (DGIST), Deagu 42988, Republic of Korea
Funding
DGIST R&D program of the Ministry of Science and ICT of KOREA · Institute of Information & communications Technology Planning & Evaluation (IITP)
References (38)
Show source-provided references
1. Singh, A., Chopra, A., Garza, E., Zhang, E., Vepakomma, P., Sharma, V., and Raskar, R. (2021, January 20–25). Disco: Dynamic and invariant sensitive channel obfuscation for deep neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01195 ↗
2. Feng, Y., Feng, Y., You, H., Zhao, X., and Gao, Y. (February, January 27). Meshnet: Mesh neural network for 3d shape representation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
3. Clark, K., Luong, M., Le, Q.V., and Manning, C.D. (2020, January 26–30). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. Available online: OpenReview.net.
4. Language models are unsupervised multitask learners
5. Amodeo, I., De Nunzio, G., Raffaeli, G., Borzani, I., Griggio, A., Conte, L., Macchini, F., Condò, V., Persico, N., and Fabietti, I. (2021). A maChine and deep Learning Approach to predict pulmoNary hyperteNsIon in newbornS with congenital diaphragmatic Hernia (CLANNISH): Protocol for a retrospective study. PLoS ONE, 16.10.1371/journal.pone.0259724 ↗
6. Taormina, V., Raso, G., Gentile, V., Abbene, L., Buttacavoli, A., Bonsignore, G., Valenti, C., Messina, P., Scardina, G.A., and Cascio, D. (2023). Automated Stabilization, Enhancement and Capillaries Segmentation in Videocapillaroscopy. Sensors, 23.10.3390/s23187674 ↗
7. Dong, J., Wang, Y., Lai, J.H., and Xie, X. (2022, January 18–24). Improving adversarially robust few-shot image classification with generalizable representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00882 ↗
8. Goodfellow, I.J., Shlens, J., and Szegedy, C. (2015, January 7–9). Explaining and Harnessing Adversarial Examples. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
9. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (May, January 30). Towards Deep Learning Models Resistant to Adversarial Attacks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
10. Kurakin, A., Goodfellow, I.J., and Bengio, S. (2018). Artificial Intelligence Safety and Security, Chapman and Hall/CRC.
11. Carlini, N., and Wagner, D. (2017, January 22–26). Towards evaluating the robustness of neural networks. Proceedings of the 2017 IEEE Symposium on Security and Privacy (sp), San Jose, CA, USA.10.1109/SP.2017.49 ↗
12. One pixel attack for fooling deep neural networks10.1109/TEVC.2019.2890858 ↗
13. Croce, F., and Hein, M. (2020, January 13–18). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event.
14. Silva, S.H., and Najafirad, P. (2020). Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv.
15. Liang, H., He, E., Zhao, Y., Jia, Z., and Li, H. (2022). Adversarial attack and defense: A survey. Electronics, 11.10.3390/electronics11081283 ↗
16. Masci, J., Meier, U., Cireşan, D., and Schmidhuber, J. (2011, January 14–17). Stacked convolutional auto-encoders for hierarchical feature extraction. Proceedings of the Artificial Neural Networks and Machine Learning–ICANN 2011: 21st International Conference on Artificial Neural Networks, Espoo, Finland.10.1007/978-3-642-21735-7_7 ↗
17. Hinton, G., Vinyals, O., and Dean, J. (2015, January 7–12). Distilling the Knowledge in a Neural Network. Proceedings of the NIPS Deep Learning and Representation Learning Workshop, Montreal, QC, Canada.
18. Kurakin, A., Goodfellow, I.J., and Bengio, S. (2016, January 2–4). Adversarial Machine Learning at Scale. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
19. Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A.L., and Le, Q.V. (2020, January 18–22). Adversarial examples improve image recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00090 ↗
20. Adversarial self-supervised contrastive learning
21. Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks
22. Song, Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. (April, January 30). PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
23. Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19–24). Pixel recurrent neural networks. Proceedings of the International Conference on Machine Learning, PMLR, New York City, NY, USA.
24. Yoon, J., Hwang, S.J., and Lee, J. (2021, January 18–24). Adversarial purification with score-based generative models. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event.
25. Generative adversarial networks10.1145/3422622 ↗
26. Samangouei, P., Kabkab, M., and Chellappa, R. (May, January 30). Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
27. Jin, G., Shen, S., Zhang, D., Dai, F., and Zhang, Y. (2019, January 12–17). Ape-gan: Adversarial perturbation elimination with gan. Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Brighton, UK.10.1109/ICASSP.2019.8683044 ↗
28. Kos, J., Fischer, I., and Song, D. (2018, January 24). Adversarial examples for generative models. Proceedings of the 2018 IEEE Security and Privacy Workshops (spw), IEEE, San Francisco, CA, USA.10.1109/SPW.2018.00014 ↗
29. Naseer, M., Khan, S., Hayat, M., Khan, F.S., and Porikli, F. (2020, January 13–19). A self-supervised approach for adversarial robustness. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00034 ↗
30. Shi, C., Holtz, C., and Mishne, G. (2020, January 30). Online Adversarial Purification based on Self-supervised Learning. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
31. Kingma, D.P., and Welling, M. (2014, January 14–16). Auto-Encoding Variational Bayes. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada.
32. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. (2008, January 5–9). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international Conference on Machine Learning, Helsinki, Finland.10.1145/1390156.1390294 ↗
33. Puvae: A variational autoencoder to purify adversarial examples10.1109/ACCESS.2019.2939352 ↗
34. Kalaria, D.R., Hazra, A., and Chakrabarti, P.P. (2022, January 9). Towards Adversarial Purification using Denoising AutoEncoders. Proceedings of the NeurIPS ML Safety Workshop, Virtual Event.
35. Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. (2022, January 17–23). Diffusion Models for Adversarial Purification. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
36. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18.
37. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
38. Imagenet classification with deep convolutional neural networks