Designing Trojan Detectors in Neural Networks Using Interactive Simulations
Source abstract
Abstract
This paper addresses the problem of designing trojan detectors in neural networks (NNs) using interactive simulations. Trojans in NNs are defined as triggers in inputs that cause misclassification of such inputs into a class (or classes) unintended by the design of a NN-based model. The goal of our work is to understand encodings of a variety of trojan types in fully connected layers of neural networks. Our approach is: (1) to simulate nine types of trojan embeddings into dot patterns; (2) to devise measurements of NN states; and (3) to design trojan detectors in NN-based classification models. The interactive simulations are built on top of TensorFlow Playground with in-memory storage of data and NN coefficients. The simulations provide analytical, visualization, and output operations performed on training datasets and NN architectures. The measurements of a NN include: (a) model inefficiency using modified Kullback–Liebler (KL) divergence from uniformly distributed states; and (b) model sensitivity to variables related to data and NNs. Using the KL divergence measurements at each NN layer and per each predicted class label, a trojan detector is devised to discriminate NN models with or without trojans. To document robustness of such a trojan detector with respect to NN architectures, dataset perturbations, and trojan types, several properties of the KL divergence measurement are presented.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Explainable Artificial Intelligence (XAI) · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Information Technology Laboratory, National Institute of Standards and Technology (NIST), 100 Bureau Drive, Gaithersburg, MD 20899, USA
National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), Axle Informatics, 6116 Executive Blvd Suite 400, Rockville, MD 20852, USA
References (39)
Show source-provided references
1. IARPA (2021, February 19). Trojans in Artificial Intelligence (TrojAI), Available online: https://pages.nist.gov/trojai/.
2. A Survey of the Recent Architectures of Deep Convolutional Neural Networks10.1007/s10462-020-09825-6 ↗
3. Trask, A., Gilmore, D., and Russell, M. (2015, January 6–11). Modeling order in neural word embeddings at scale. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
4. Justus, D., Brennan, J., Bonner, S., and McGough, A.S. (2019). Predicting the Computational Cost of Deep Learning Models. arXiv.10.1109/BigData.2018.8622396 ↗
5. Doran, D., Schulz, S., and Besold, T.R. (2018). What does explainable AI really mean? A new conceptualization of perspectives. arXiv.
6. Bruna, J., and Dec, L.G. (2021, February 19). Mathematics of Deep Learning. Available online: https://arxiv.org/pdf/1712.04741.pdf.
7. Unser, M. (2021, February 19). A Representer Theorem for Deep Neural Networks. Available online: https://arxiv.org/pdf/1802.09210.pdf.
8. Understanding Deep Convolutional Networks
9. Smilkov, D., Carter, S., Sculley, D., Viégas, F.B., and Wattenberg, M. (2017). Direct-Manipulation Visualization of Deep Networks. arXiv.
10. Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. (2017, January 4–9). The expressive power of neural networks: A view from the width. Proceedings of the Internet Society, Advances in Neural Information Processing Systems, Long Beach, CA, USA.
11. Schaub, N.J., and Hotaling, N. (2020). Assessing Intelligence in Artificial Neural Networks. arXiv.
12. Zhao, B., and Lao, Y. (2018, January 22–24). Resilience of Pruned Neural Network Against Poisoning Attack. Proceedings of the 13th International Conference on Malicious and Unwanted Software (MALWARE), Nantucket, MA, USA.10.1109/MALWARE.2018.8659362 ↗
13. Siegelmann, H. (2021, February 19). Guaranteeing AI Robustness against Deception (GARD). Available online: https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception.
14. Xu, X., Wang, Q., Li, H., Borisov, N., Gunter, C.A., and Li, B. (2019). Detecting AI Trojans Using Meta Neural Analysis. arXiv.
15. Roth, K., Kilcher, Y., and Hofmann, T. (2019). The Odds are Odd: A Statistical Test for Detecting Adversarial Examples. arXiv.
16. Liu, Y., Ma, S., Aafer, Y., Lee, W.C., Zhai, J., Wang, W., and Zhang, X. (2018, January 18–21). Trojaning Attack on Neural Networks. Proceedings of the Internet Society, Network and Distributed Systems Security (NDSS) Symposium 2018, San Diego, CA, USA.10.14722/ndss.2018.23291 ↗
17. Fine-pruning: Defending against backdooring attacks on deep neural networks
18. Tan, T.J.L., and Shokri, R. (2019). Bypassing Backdoor Detection Algorithms in Deep Learning. arXiv.10.1109/EuroSP48549.2020.00019 ↗
19. LeCun, Y., Denker, J.S., and Solla, S.A. (1989). Optimal Brain Damage. Proceedings of the Neural Information Processing Systems, AT&T Bell Laboratory, Neural Information Processing Systems Foundation, Inc.
20. Hassibi, B., and Stork, D.G. (1992). Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. Advances in Neural Information Processing Systems 5 (NIPS 1992), Neural Information Processing Systems Foundation, Inc.
21. Hu, H., Peng, R., Tai, Y.w., Limited, S.G., and Tang, C.k. (2016). Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures. arXiv.
22. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2017, January 24–26). Pruning Filters for Efficient ConvNets. Proceedings of the International Conference on Learning Representations, Toulon, France.
23. Han, S., Pool, J., Tran, J., and Dally, W.J. (2015). Learning both Weights and Connections for Efficient Neural Networks. arXiv.
24. Reconciling modern machine learning practice and the bias-variance trade-off10.1073/pnas.1903070116 ↗
25. Zeiler, M.D., and Fergus, R. (2013). Visualizing and Understanding Convolutional Networks. arXiv.10.1007/978-3-319-10590-1_53 ↗
26. Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network, University of Montreal. Technical Report.
27. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2015). Learning Deep Features for Discriminative Localization. arXiv.10.1109/CVPR.2016.319 ↗
28. Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J. (2016). Quantized Convolutional Neural Networks for Mobile Devices. arXiv.10.1109/CVPR.2016.521 ↗
29. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). XNOR-Net: ImageNet Classification Using Binary. arXiv.10.1007/978-3-319-46493-0_32 ↗
30. Gupta, S., Agrawal, A., Gopalakrishnan, K., Heights, Y., Narayanan, P., and Jose, S. (2021, February 19). Deep Learning with Limited Numerical Precision. Available online: http://proceedings.mlr.press/v37/gupta15.pdf.
31. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016, January 5–10). Binarized Neural Networks. Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
32. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
33. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015-Conference Track Proceedings, San Diego, CA, USA.
34. Novak, R., Bahri, Y., Abolafia, D.A., Pennington, J., and Sohl-dickstein, J. (2018). Sensitivity and Generalization in Neural Networks: An Empirical Study. arXiv.
35. Shwartz-Ziv, R., Painsky, A., and Tishby, N. (2019, January 6–9). Representation Compression and Generalization in Deep Neural Networks. Proceedings of the The International Conference on Learning Representations (ICLR), New Orleans, LA, USA.
36. Approximation capabilities of multilayer feedforward networks10.1016/0893-6080(91)90009-T ↗
37. A Mathematical Theory of Communication10.1002/j.1538-7305.1948.tb01338.x ↗
38. On Information and Sufficiency10.1214/aoms/1177729694 ↗
39. Nielsen, F. (2010). A Family of Statistical Symmetric Divergences Based On Jensen’s Inequality. arXiv.