Autonomous Penetration Testing Based on Improved Deep Q-Network
Source abstract
Abstract
Penetration testing is an effective way to test and evaluate cybersecurity by simulating a cyberattack. However, the traditional methods deeply rely on domain expert knowledge, which requires prohibitive labor and time costs. Autonomous penetration testing is a more efficient and intelligent way to solve this problem. In this paper, we model penetration testing as a Markov decision process problem and use reinforcement learning technology for autonomous penetration testing in large scale networks. We propose an improved deep Q-network (DQN) named NDSPI-DQN to address the sparse reward problem and large action space problem in large-scale scenarios. First, we reasonably integrate five extensions to DQN, including noisy nets, soft Q-learning, dueling architectures, prioritized experience replay, and intrinsic curiosity model to improve the exploration efficiency. Second, we decouple the action and split the estimators of the neural network to calculate two elements of action separately, so as to decrease the action space. Finally, the performance of algorithms is investigated in a range of scenarios. The experiment results demonstrate that our methods have better convergence and scaling performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Malware Detection Techniques · Adversarial Robustness in Machine Learning · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronic Engineering, National University of Defense Technology, Hefei 230000, China
Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation, Hefei 230037, China
References (31)
Show source-provided references
1. Attack trees
2. Phillips, C., and Swiler, L.P. (1998, January 22–26). A graph-based system for network-vulnerability analysis. Proceedings of the 1998 Workshop on New Security Paradigms, Charlottesville, VA, USA.10.1145/310889.310919 ↗
3. Sarraute, C., Buffet, O., and Hoffmann, J. (2012, January 22–26). POMDPs make better hackers: Accounting for uncertainty in penetration testing. Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, ON, Canada.10.1609/aaai.v26i1.8363 ↗
4. Sarraute, C., Buffet, O., and Hoffmann, J. (2013). Penetration testing== pomdp solving?. arXiv.
5. Hoffmann, J. (2015, January 7–11). Simulated Penetration Testing: From “Dijkstra” to “Turing Test++”. Proceedings of the Twenty-Fifth International Conference on Automated Planning and Scheduling, Jerusalem, Israel.10.1609/icaps.v25i1.13684 ↗
6. Schwartz, J., Kurniawati, H., and El-Mahassni, E. (2020, January 14–19). POMDP+ Information-Decay: Incorporating Defender’s Behaviour in Autonomous Penetration Testing. Proceedings of the International Conference on Automated Planning and Scheduling, Nancy, France.10.1609/icaps.v30i1.6666 ↗
7. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
8. Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2019). Dota 2 with large scale deep reinforcement learning. arXiv.
9. Grandmaster level in StarCraft II using multi-agent reinforcement learning10.1038/s41586-019-1724-z ↗
10. Ye, D., Chen, G., Zhang, W., Chen, S., Yuan, B., Liu, B., Chen, J., Liu, Z., Qiu, F., and Yu, H. (2020). Towards playing full moba games with deep reinforcement learning. arXiv.10.1609/aaai.v34i04.6144 ↗
11. Canese, L., Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., and Spanò, S. (2021). Multi-Agent Reinforcement Learning: A Review of Challenges and Applications. Appl. Sci., 11.10.3390/app11114948 ↗
12. Zennaro, F.M., and Erdodi, L. (2020). Modeling penetration testing with reinforcement learning using capture-the-flag challenges and tabular Q-learning. arXiv.
13. Erdodi, L., Sommervoll, Å.Å., and Zennaro, F.M. (2021). Simulating SQL Injection Vulnerability Exploitation Using Q-Learning Reinforcement Learning Agents. arXiv.10.1016/j.jisa.2021.102903 ↗
14. Chowdhary, A., Huang, D., Mahendran, J.S., Romo, D., Deng, Y., and Sabur, A. (2020, January 17–19). Autonomous security analysis and penetration testing. Proceedings of the 2020 16th International Conference on Mobility, Sensing and Networking (MSN), Tokyo, Japan.10.1109/MSN50589.2020.00086 ↗
15. Schwartz, J., and Kurniawati, H. (2019). Autonomous penetration testing using reinforcement learning. arXiv.
16. Seifert, C., Betser, M., Blum, W., Bono, J., Farris, K., Goren, E., Grana, J., Holsheimer, K., Marken, B., and Neil, J. (2021, September 06). CyberBattleSim. Available online: https://github.com/microsoft/cyberbattlesim.
17. Survey on Sparse Reward in Deep reinforcement Learning
18. Schwartz, J., and Kurniawatti, H. (2021, September 06). NASim: Network Attack Simulator. Available online: https://networkattacksimulator.readthedocs.io/.
19. Strom, B.E., Applebaum, A., Miller, D.P., Nickels, K.C., Pennington, A.G., and Thomas, C.B. (2018). Mitre att&ck: Design and philosophy. Tech. Rep.
20. A Markovian decision process
21. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
22. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
23. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
24. Reward is enough10.1016/j.artint.2021.103535 ↗
25. Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. (2017, January 6–11). Reinforcement learning with deep energy-based policies. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
26. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016, January 20–22). Dueling network architectures for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
27. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized experience replay. arXiv.
28. Fortunato, M., Azar, M.G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., and Pietquin, O. (2017). Noisy networks for exploration. arXiv.
29. Pathak, D., Agrawal, P., Efros, A.A., and Darrell, T. (2017, January 6–11). Curiosity-driven exploration by self-supervised prediction. Proceedings of the International Conference on Machine Learning, Sydney, Australia.10.1109/CVPRW.2017.70 ↗
30. Speicher, P., Steinmetz, M., Hoffmann, J., Backes, M., and Künnemann, R. (2019, January 8–12). Towards automated network mitigation analysis. Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, Limassol, Cyprus.10.1145/3297280.3297473 ↗
31. Backes, M., Hoffmann, J., Künnemann, R., Speicher, P., and Steinmetz, M. (2017). Simulated penetration testing and mitigation analysis. arXiv.