Review of Artificial Intelligence Adversarial Attack and Defense Technologies
Source abstract
Abstract
In recent years, artificial intelligence technologies have been widely used in computer vision, natural language processing, automatic driving, and other fields. However, artificial intelligence systems are vulnerable to adversarial attacks, which limit the applications of artificial intelligence (AI) technologies in key security fields. Therefore, improving the robustness of AI systems against adversarial attacks has played an increasingly important role in the further development of AI. This paper aims to comprehensively summarize the latest research progress on adversarial attack and defense technologies in deep learning. According to the target model’s different stages where the adversarial attack occurred, this paper expounds the adversarial attack methods in the training stage and testing stage respectively. Then, we sort out the applications of adversarial attack technologies in computer vision, natural language processing, cyberspace security, and the physical world. Finally, we describe the existing adversarial defense methods respectively in three main categories, i.e., modifying data, modifying models and using auxiliary tools.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Anomaly Detection Techniques and Applications · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The School of Information and Software Enginerring, University of Electronic Science and Technology of China, Chengdu 610054, China
References (91)
Show source-provided references
1. Deep neural nets as a method for quantitative structure–activity relationships10.1021/ci500747n ↗
2. Connectomic reconstruction of the inner plexiform layer in the mouse retina10.1038/nature12346 ↗
3. Online particle detection with neural networks based on topological calorimetry information10.1088/1742-6596/368/1/012030 ↗
4. Adam-Bourdarios, C., Cowan, G., Germain, C., Guyon, I., Kégl, B., and Rousseau, D. (2014, January 8–13). The Higgs boson machine learning challenge. Proceedings of the NIPS 2014 Workshop on High-Energy Physics and Machine Learning, Montreal, QC, Canada.
5. The human splicing code reveals new insights into the genetic determinants of disease10.1126/science.1254806 ↗
6. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (arXiv, 2013). Intriguing properties of neural networks, arXiv.
7. Barreno, M., Nelson, B., Sears, R., Joseph, A.D., and Tygar, J.D. (2006, January 21–24). Can machine learning be secure?. Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, Taipei, Taiwan.10.1145/1128817.1128824 ↗
8. Biggio, B., Nelson, B., and Laskov, P. (2011, January 13–15). Support vector machines under adversarial label noise. Proceedings of the Asian Conference on Machine Learning, Taoyuan, Taiwan.
9. Kloft, M., and Laskov, P. (2010, January 13–15). Online anomaly detection under adversarial impact. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy.
10. Security analysis of online centroid anomaly detection
11. Biggio, B., Nelson, B., and Laskov, P. (arXiv, 2012). Poisoning attacks against support vector machines, arXiv.
12. Biggio, B., Didaci, L., Fumera, G., and Roli, F. (2013, January 4–7). Poisoning attacks to compromise face templates. Proceedings of the 2013 International Conference on Biometrics (ICB), Madrid, Spain.10.1109/ICB.2013.6613006 ↗
13. Mei, S., and Zhu, X. (2015, January 25–30). Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9569 ↗
14. Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016, January 22–26). Distillation as a defense to adversarial perturbations against deep neural networks. Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.10.1109/SP.2016.41 ↗
15. Goodfellow, I.J., Shlens, J., and Szegedy, C. (arXiv, 2014). Explaining and harnessing adversarial examples, arXiv.
16. Kurakin, A., Goodfellow, I., and Bengio, S. (arXiv, 2016). Adversarial machine learning at scale, arXiv.
17. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., and Swami, A. (2016, January 21–24). The limitations of deep learning in adversarial settings. Proceedings of the 2016 IEEE European Symposium on Security and Privacy (EuroS&P), Saarbrucken, Germany.10.1109/EuroSP.2016.36 ↗
18. Su, J., Vargas, D.V., and Kouichi, S. (arXiv, 2017). One pixel attack for fooling deep neural networks, arXiv.
19. Moosavi-Dezfooli, S.M., Fawzi, A., and Frossard, P. (2016, January 27–30). Deepfool: A simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.282 ↗
20. Cisse, M., Adi, Y., Neverova, N., and Keshet, J. (arXiv, 2017). Houdini: Fooling deep structured prediction models, arXiv.
21. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., and Swami, A. (2017, January 2–6). Practical black-box attacks against machine learning. Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, Abu Dhabi, UAE.10.1145/3052973.3053009 ↗
22. Fredrikson, M., Jha, S., and Ristenpart, T. (2015, January 12–16). Model inversion attacks that exploit confidence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA.10.1145/2810103.2813677 ↗
23. Tramèr, F., Zhang, F., Juels, A., Reiter, M.K., and Ristenpart, T. (2016, January 10–12). Stealing Machine Learning Models via Prediction APIs. Proceedings of the USENIX Security Symposium, Austin, TX, USA.
24. Carlini, N., and Wagner, D. (2017, January 22–26). Towards evaluating the robustness of neural networks. Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.10.1109/SP.2017.49 ↗
25. Chen, P.Y., Zhang, H., Sharma, Y., Yi, J., and Hsieh, C.J. (2017, January 3). Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, Dallas, TX, USA.10.1145/3128572.3140448 ↗
26. Dong, Y., Liao, F., Pang, T., Su, H., Hu, X., Li, J., and Zhu, J. (arXiv, 2017). Boosting adversarial attacks with momentum, arXiv.10.1109/CVPR.2018.00957 ↗
27. Xiao, C., Deng, R., Li, B., Yu, F., Liu, M., and Song, D. (arXiv, 2018). Characterizing adversarial examples based on spatial consistency information for semantic segmentation, arXiv.10.1007/978-3-030-01249-6_14 ↗
28. Wei, X., Liang, S., Cao, X., and Zhu, J. (arXiv, 2018). Transferable Adversarial Attacks for Image and Video Object Detection, arXiv.10.24963/ijcai.2019/134 ↗
29. Belinkov, Y., and Bisk, Y. (arXiv, 2017). Synthetic and natural noise both break neural machine translation, arXiv.
30. Liang, B., Li, H., Su, M., Bian, P., Li, X., and Shi, W. (arXiv, 2017). Deep text classification can be fooled, arXiv.10.24963/ijcai.2018/585 ↗
31. Katz, G., Barrett, C., Dill, D.L., Julian, K., and Kochenderfer, M.J. (arXiv, 2017). Towards proving the adversarial robustness of deep neural networks, arXiv.10.4204/EPTCS.257.3 ↗
32. Krotov, D., and Hopfield, J.J. (arXiv, 2017). Dense associative memory is robust to adversarial inputs, arXiv.10.1162/neco_a_01143 ↗
33. Grosse, K., Papernot, N., Manoharan, P., Backes, M., and McDaniel, P. (arXiv, 2016). Adversarial perturbations against deep neural networks for malware classification, arXiv.10.1109/SP.2016.41 ↗
34. Huang, C.H., Lee, T.H., Chang, L.H., Lin, J.R., and Horng, G. (2018). Adversarial Attacks on SDN-Based Deep Learning IDS System. International Conference on Mobile and Wireless Technology, Springer.10.1007/978-981-13-1059-1_17 ↗
35. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., and Song, D. (2018). Robust physical-world attacks on deep learning visual classification. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.10.1109/CVPR.2018.00175 ↗
36. Melis, M., Demontis, A., Biggio, B., Brown, G., Fumera, G., and Roli, F. (2017, January 22–29). Is deep learning safe for robot vision? adversarial examples against the icub humanoid. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshop (ICCVW), Venice, Italy.10.1109/ICCVW.2017.94 ↗
37. Sharif, M., Bhagavatula, S., Bauer, L., and Reiter, M.K. (2016, January 24–28). Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria.10.1145/2976749.2978392 ↗
38. Zhou, Z., Tang, D., Wang, X., Han, W., Liu, X., and Zhang, K. (arXiv, 2018). Invisible Mask: Practical Attacks on Face Recognition with Infrared, arXiv.
39. Yann, L., Corinna, C., and Christopher, J. (2017, May 06). MNIST. Available online: http://yann.lecun.com/exdb/mnist/.
40. Huang, R., Xu, B., Schuurmans, D., and Szepesvári, C. (arXiv, 2015). Learning with a strong adversary, arXiv.
41. Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P. (arXiv, 2017). Ensemble adversarial training: Attacks and defenses, arXiv.
42. Hosseini, H., Chen, Y., Kannan, S., Zhang, B., and Poovendran, R. (arXiv, 2017). Blocking transferability of adversarial examples in black-box learning systems, arXiv.
43. Dziugaite, G.K., Ghahramani, Z., and Roy, D.M. (arXiv, 2016). A study of the effect of jpg compression on adversarial images, arXiv.
44. Das, N., Shanbhogue, M., Chen, S.T., Hohman, F., Chen, L., Kounavis, M.E., and Chau, D.H. (arXiv, 2017). Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg compression, arXiv.
45. Xie, C., Wang, J., Zhang, Z., Zhou, Y., Xie, L., and Yuille, A. (arXiv, 2017). Adversarial examples for semantic segmentation and object detection, arXiv.10.1109/ICCV.2017.153 ↗
46. Wang, Q., Guo, W., Zhang, K., Ororbia, I., Alexander, G., Xing, X., Liu, X., and Giles, C.L. (arXiv, 2016). Learning adversary-resistant deep neural networks, arXiv.
47. Lyu, C., Huang, K., and Liang, H.N. (2015, January 14–17). A unified gradient regularization family for adversarial examples. Proceedings of the 2015 IEEE International Conference on Data Mining (ICDM), Atlantic City, NJ, USA.10.1109/ICDM.2015.84 ↗
48. Zhao, Q., and Griffin, L.D. (arXiv, 2016). Suppressing the unusual: Towards robust cnns using symmetric activation functions, arXiv.
49. Rozsa, A., Gunther, M., and Boult, T.E. (arXiv, 2016). Towards robust deep neural networks with BANG, arXiv.
50. Papernot, N., and McDaniel, P. (arXiv, 2017). Extending defensive distillation, arXiv.
51. Xu, W., Evans, D., and Qi, Y. (arXiv, 2017). Feature squeezing: Detecting adversarial examples in deep neural networks, arXiv.10.14722/ndss.2018.23198 ↗
52. Gu, S., and Rigazio, L. (arXiv, 2014). Towards deep neural network architectures robust to adversarial examples, arXiv.
53. Gao, J., Wang, B., Lin, Z., Xu, W., and Qi, Y. (arXiv, 2017). Deepcloak: Masking deep neural network models for robustness against adversarial samples, arXiv.
54. Samangouei, P., Kabkab, M., and Chellappa, R. (arXiv, 2018). Defense-GAN: Protecting classifiers against adversarial attacks using generative models, arXiv.
55. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–10 December 2013, Curran Associates, Inc.
56. Meng, D., and Chen, H. (November, January 30). Magnet: A two-pronged defense against adversarial examples. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3134057 ↗
57. Liao, F., Liang, M., Dong, Y., Pang, T., Zhu, J., and Hu, X. (arXiv, 2017). Defense against adversarial attacks using high-level representation guided denoiser, arXiv.10.1109/CVPR.2018.00191 ↗
58. Taga, K., Kameyama, K., and Toraichi, K. (2003, January 28–30). Regularization of hidden layer unit response for neural networks. Proceedings of the 2003 IEEE Pacific Rim Conference on Communications, Computers and signal Processing, Victoria, BC, Canada.
59. Zhang, J., and Jiang, X. (arXiv, 2018). Adversarial Examples: Opportunities and Challenges, arXiv.
60. Learning in the presence of malicious errors10.1137/0222052 ↗
61. Miyato, T., Maeda, S.i., Ishii, S., and Koyama, M. (2018). Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2018.2858821 ↗
62. Papernot, N., McDaniel, P., and Goodfellow, I. (arXiv, 2016). Transferability in machine learning: From phenomena to black-box attacks using adversarial samples, arXiv.
63. Fredrikson, M., Lantz, E., Jha, S., Lin, S., Page, D., and Ristenpart, T. (2014, January 20–22). Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing. Proceedings of the USENIX Security Symposium, San Diego, CA, USA.
64. Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., and Frossard, P. (arXiv, 2017). Universal adversarial perturbations, arXiv.10.1109/CVPR.2017.17 ↗
65. Sarkar, S., Bansal, A., Mahbub, U., and Chellappa, R. (arXiv, 2017). UPSET and ANGRI: Breaking High Performance Image Classifiers, arXiv.
66. Baluja, S., and Fischer, I. (arXiv, 2017). Adversarial transformation networks: Learning to generate adversarial examples, arXiv.10.1609/aaai.v32i1.11672 ↗
67. Ilyas, A., Engstrom, L., Athalye, A., and Lin, J. (arXiv, 2018). Black-box Adversarial Attacks with Limited Queries and Information, arXiv.
68. Li, P., Yi, J., and Zhang, L. (arXiv, 2018). Query-Efficient Black-Box Attack by Active Learning, arXiv.
69. Adate, A., and Saxena, R. (2017, January 20–22). Understanding How Adversarial Noise Affects Single Image Classification. Proceedings of the International Conference on Intelligent Information Technologies, Chennai, India.10.1007/978-981-10-7635-0_22 ↗
70. Zhang, X., Zhao, J., and LeCun, Y. (2016). Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems10.18653/v1/D16-1076 ↗
71. Proceedings of the Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 7-12 December 2015, Curran Associates, Inc.
72. Kulynych, B. (2017, May 06). TextFool. Available online: https://github.com/bogdan-kulynych/textfool.
73. Hosseini, H., Kannan, S., Zhang, B., and Poovendran, R. (arXiv, 2017). Deceiving Google’s Perspective API Built for Detecting Toxic Comments, arXiv.
74. Samanta, S., and Mehta, S. (2018). Generating Adversarial Text Samples. Advances in Information Retrieval, Proceedings of the 40th European Conference on Information Retrieval Research, Grenoble, France, 26–29 March 2018, Springer International Publishing.10.1007/978-3-319-76941-7_71 ↗
75. Zhao, Z., Dua, D., and Singh, S. (arXiv, 2017). Generating natural adversarial examples, arXiv.
76. Ebrahimi, J., Lowd, D., and Dou, D. (arXiv, 2018). On Adversarial Examples for Character-Level Neural Machine Translation, arXiv.
77. Liu, Y., Chen, X., Liu, C., and Song, D. (arXiv, 2016). Delving into transferable adversarial examples and black-box attacks, arXiv.
78. Hu, W., and Tan, Y. (arXiv, 2017). Generating adversarial malware examples for black-box attacks based on GAN, arXiv.
79. Rosenberg, I., Shabtai, A., Rokach, L., and Elovici, Y. (2018, January 10–12). Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers. Proceedings of the International Symposium on Research in Attacks, Intrusions, and Defenses, Heraklion, Greece.10.1007/978-3-030-00470-5_23 ↗
80. Grosse, K., Papernot, N., Manoharan, P., Backes, M., and McDaniel, P. (2017, January 11–15). Adversarial examples for malware detection. Proceedings of the European Symposium on Research in Computer Security, Oslo, Norway.10.1007/978-3-319-66399-9_4 ↗
81. Threat of adversarial attacks on deep learning in computer vision: A survey10.1109/ACCESS.2018.2807385 ↗
82. Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey10.1109/TITS.2012.2209421 ↗
83. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition10.1016/j.neunet.2012.02.016 ↗
84. Gu, T., Dolan-Gavitt, B., and Garg, S. (arXiv, 2017). Badnets: Identifying vulnerabilities in the machine learning model supply chain, arXiv.
85. Xu, X., Chen, X., Liu, C., Rohrbach, A., Darell, T., and Song, D. (arXiv, 2017). Can you fool AI with adversarial examples on a visual Turing test?, arXiv.
86. Rinaldi (2011, May 06). Actress Reese Witherspoon. By Eva Rinaldi/CC BY-SA/cropped. Available online: https://goo.gl/a2sCdc.
87. (2012, December 21). Rinaldi The Target. Eva Rinaldi/CC BY-SA/Cropped. Available online: https://goo.gl/AO7QYu.
88. Akhtar, N., Liu, J., and Mian, A. (arXiv, 2017). Defense against Universal Adversarial Perturbations, arXiv.10.1109/CVPR.2018.00357 ↗
89. Hinton, G., Vinyals, O., and Dean, J. (arXiv, 2015). Distilling the knowledge in a neural network, arXiv.
90. Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Y. (July, January 28). Contractive auto-encoders: Explicit invariance during feature extraction. Proceedings of the 28th International Conference on International Conference on Machine Learning, Bellevue, WA, USA.
91. An introduction to frames