Cascaded Reinforcement Learning Agents for Large Action Spaces in Autonomous Penetration Testing
Source abstract
Abstract
Organised attacks on a computer system to test existing defences, i.e., penetration testing, have been used extensively to evaluate network security. However, penetration testing is a time-consuming process. Additionally, establishing a strategy that resembles a real cyber-attack typically requires in-depth knowledge of the cybersecurity domain. This paper presents a novel architecture, named deep cascaded reinforcement learning agents, or CRLA, that addresses large discrete action spaces in an autonomous penetration testing simulator, where the number of actions exponentially increases with the complexity of the designed cybersecurity network. Employing an algebraic action decomposition strategy, CRLA is shown to find the optimal attack policy in scenarios with large action spaces faster and more stably than a conventional deep Q-learning agent, which is commonly used as a method for applying artificial intelligence to autonomous penetration testing.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Advanced Malware Detection Techniques · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia
Defence Science and Technology Group, Edinburgh, SA 5111, Australia
Funding
Australian Defence Science Technology Group (DSTG)
References (26)
Show source-provided references
1. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
2. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
3. Q-learning10.1007/BF00992698 ↗
4. Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. (2015). Dueling network architectures for deep reinforcement learning. arXiv.
5. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.G., and Silver, D. (2018, January 2–7). Rainbow: Combining Improvements in Deep Reinforcement Learning. Proceedings of the Thirty-second AAAI conference on artificial intelligence 2018, New Orleans, LA, USA.10.1609/aaai.v32i1.11796 ↗
6. A Theoretical Analysis of Deep Q-Learning
7. Farquhar, G., Gustafson, L., Lin, Z., Whiteson, S., Usunier, N., and Synnaeve, G. (2020, January 24–26). Growing Action Spaces. Proceedings of the International Conference on Machine Learning 2020, Virtual.
8. Schwartz, J., and Kurniawati, H. (2019). Autonomous Penetration Testing using Reinforcement Learning. arXiv.
9. Tavakoli, A., Pardo, F., and Kormushev, P. (2018, January 2–7). Action branching architectures for deep reinforcement learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11798 ↗
10. Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv.
11. Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., and Marriott, D. (2021). CybORG: A Gym for the Development of Autonomous Cyber Agents. arXiv.
12. Hu, Z., Beuran, R., and Tan, Y. (2020, January 7–11). Automated Penetration Testing Using Deep Reinforcement Learning. Proceedings of the 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), Genoa, Italy.10.1109/EuroSPW51379.2020.00010 ↗
13. Zennaro, F.M., and Erdodi, L. (2020). Modeling penetration testing with reinforcement learning using capture-the-flag challenges and tabular Q-learning. arXiv.
14. Zahavy, T., Haroush, M., Merlis, N., Mankowitz, D.J., and Mannor, S. (2018, January 3–8). Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.10.1109/INDIANCC.2019.8715603 ↗
15. Chandak, Y., Theocharous, G., Kostas, J., Jordan, S.M., and Thomas, P.S. (2019, January 10–15). Learning Action Representations for Reinforcement Learning. Proceedings of the International Conference on Machine Learning 2019, Long Beach, CA, USA.
16. de Wiele, T.V., Warde-Farley, D., Mnih, A., and Mnih, V. (2020). Q-Learning in enormous action spaces via amortized approximate maximization. arXiv.
17. Rashid, T., Samvelyan, M., de Witt, C.S., Farquhar, G., Foerster, J.N., and Whiteson, S. (2018, January 10–15). QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Proceedings of the International Conference on Machine Learning 2018, Stockholm, Sweden.
18. Masson, W., and Konidaris, G.D. (2016, January 12–17). Reinforcement Learning with Parameterized Actions. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence 2016, Phoenix, AZ, USA.10.1609/aaai.v30i1.10226 ↗
19. Fan, Z., Su, R., Zhang, W., and Yu, Y. (2019). Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space. arXiv.10.24963/ijcai.2019/316 ↗
20. Tan, M. (1993, January 27–29). Multi-Agent Reinforcement Learning: Independent versus Cooperative Agents. Proceedings of the Tenth International Conference on Machine Learning (ICML 1993), Amherst, MA, USA.10.1016/B978-1-55860-307-3.50049-6 ↗
21. Leibo, J.Z., Zambaldi, V.F., Lanctot, M., Marecki, J., and Graepel, T. (2017). Multi-agent Reinforcement Learning in Sequential Social Dilemmas. arXiv.
22. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V.F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., and Tuyls, K. (2017). Value-Decomposition Networks For Cooperative Multi-Agent Learning. arXiv.
23. Ha, D., Dai, A.M., and Le, Q.V. (2016). HyperNetworks. arXiv.
24. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 9). Automatic differentiation in PyTorch. Proceedings of the NIPS 2017 Autodiff Workshop, Long Beach, CA, USA.
25. Ghanem, M.C., and Chen, T.M. (2020). Reinforcement Learning for Efficient Network Penetration Testing. Information, 11.10.3390/info11010006 ↗
26. Visualizing data using t-SNE