Multi-Agent Framework Utilizing Large Language Models for Solving Capture-the-Flag Challenges in Cybersecurity Competitions
Source abstract
Abstract
Capture the Flag (CTF) is an important form of competition in cybersecurity, which tests participants’ knowledge and problem-solving abilities. We propose a multi-agent framework based on large language models to simulate human participants and attempt to automate the solutions of common CTF problems, especially in cryptographic and miscellaneous challenges. We implement the collaboration of multiple expert agents and access external tools to give the language model a basic level of practical competence in the field of cybersecurity. We primarily test two capabilities of the large model: to analyze, reason, and determine solutions to CTF problems, and to assist with problem-solving by generating code or utilizing unannotated existing external tools. We construct a benchmark based on the puzzles from the book “Ghost in the Wires” and the THUCTF competition. The experiment results showed that our agents performed well on the former and were significantly improved with some human hints, compared with related work. We also discuss the challenges that language models face in cybersecurity challenges and the effect of leveraging reasoning models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multi-Agent Systems and Negotiation · Information and Cyber Security · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Weiyang College, Tsinghua University, Beijing 100084, China
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
Institute for Network Sciences and Cyberspace, Tsinghua University, Beijing 100084, China
Funding
National Key R&D Program
References (40)
Show source-provided references
1. A CTF-Based Approach in Cyber Security Education for Secondary School Students
2. McDaniel, L., Talvi, E., and Hay, B.N. (2016, January 5–8). Capture the Flag as Cyber Security Introduction. Proceedings of the 2016 49th Hawaii International Conference on System Sciences (HICSS), Koloa, HI, USA.10.1109/HICSS.2016.677 ↗
3. Using Facebook’s Open Source Capture the Flag Platform as a Hands-on Learning and Assessment Tool for Cybersecurity Education
4. Leune, K., and Petrilli, S.J. (2017, January 4–7). Using Capture-the-Flag to Enhance the Effectiveness of Cybersecurity Education. Proceedings of the 18th Annual Conference on Information Technology Education, New York, NY, USA.10.1145/3125659.3125686 ↗
5. Vykopal, J., Švábenský, V., and Chang, E.C. (2020, January 11–14). Benefits and Pitfalls of Using Capture the Flag Games in University Courses. Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Portland, OR, USA.10.1145/3328778.3366893 ↗
6. The Role of CTF (Capture the FLAG) Competitions in Increasing Students’ Interest in Cyber Security: An Innovative Approach in Education
7. Venkatagiri, S., Mukhopadhyay, A., Hicks, D., Brantly, A.F., and Luther, K. (2023, January 10–14). CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation. Proceedings of the 2023 ACM Designing Interactive Systems Conference, Pittsburgh, PA, USA.10.1145/3563657.3595997 ↗
8. Chang, S.Y., Yoon, K., Wuthier, S., and Zhang, K. (2022). Capture the Flag for Team Construction in Cybersecurity. arXiv.
9. Enhancing Competency of Cybersecurity Through Implementation of the “CAPTURE THE FLAG” On College in Indonesia
10. Renaud, K., Warkentin, M., and Westerman, G. (2023). From ChatGPT to HackGPT: Meeting the Cybersecurity Threat of Generative AI, MIT Sloan Management Review.
11. Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., and Rass, S. (2023). PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. arXiv.
12. Yu, N., Davies, L., and Zhong, W. (2024, January 15–18). A Dual-Approach Framework for Enhancing Network Traffic Analysis (DAFENTA): Leveraging NumeroLogic LLM Embeddings and Transformer Models for Intrusion Detection. Proceedings of the 2024 IEEE International Conference on Big Data (BigData), Washington, DC, USA.10.1109/BigData62323.2024.10825828 ↗
13. Cui, T., Lin, X., Li, S., Chen, M., Yin, Q., Li, Q., and Xu, K. (2025). TrafficLLM: Enhancing Large Language Models for Network Traffic Analysis with Generic Traffic Representation. arXiv.
14. Adjewa, F., Esseghir, M., and Merghem-Boulahia, L. (2024). LLM-based Continuous Intrusion Detection Framework for Next-Gen Networks. arXiv.
15. Smart Contract Vulnerability Detection: The Role of Large Language Model (LLM)10.1145/3687251.3687253 ↗
16. GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning10.1016/j.jss.2024.112031 ↗
17. Xia, C., and Zhang, L. (2023, January 17–21). Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. Proceedings of the International Symposium on Software Testing and Analysis, Seattle, WA, USA.10.1145/3650212.3680323 ↗
18. Xia, C., and Zhang, L. (2023). Conversational Automated Program Repair. arXiv.
19. Exploring the potential of large language models for improving digital forensic investigation efficiency
20. Meng, X., Srivastava, A., Arunachalam, A., Ray, A., Silva, P.H., Psiakis, R., Makris, Y., and Basu, K. (2023). Unlocking Hardware Security Assurance: The Potential of LLMs. arXiv.
21. Xian, Y., Lampert, C.H., Schiele, B., and Akata, Z. (2020). Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly. arXiv.10.1109/TPAMI.2018.2857768 ↗
22. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E.H., Xia, F., Le, Q., and Zhou, D. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv.
23. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W.-T., and Rocktäschel, T. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv.
24. Paul, R., Hossain, M.M., Siddiq, M.L., Hasan, M., Iqbal, A., and Santos, J.C.S. (2023). Enhancing Automated Program Repair through Fine-tuning and Prompt Engineering. arXiv.
25. Ali, T., and Kostakos, P. (2023). HuntGPT: Integrating Machine Learning-Based Anomaly Detection and Explainable AI with Large Language Models (LLMs). arXiv.
26. Shao, M., Chen, B., Jancheska, S., Dolan-Gavitt, B., Garg, S., Karri, R., and Shafique, M. (2024). An Empirical Evaluation of LLMs for Solving Offensive Security Challenges. arXiv.
27. Muzsai, L., Imolai, D., and Lukács, A. (2024). HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing. arXiv.
28. Udeshi, M., Shao, M., Xi, H., Rani, N., Milner, K., Putrevu, V.S.C., Dolan-Gavitt, B., Shukla, S.K., Krishnamurthy, P., and Khorrami, F. (2025). D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security. arXiv.
29. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The Llama 3 Herd of Models. arXiv.
30. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., and Fu, Z. (2024). DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv.
31. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv.
32. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv.
33. Wang, B., Fang, H., Eisner, J., Durme, B.V., and Su, Y. (2024, January 11–16). LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2024.acl-long.570 ↗
34. Tran, K.T., Dao, D., Nguyen, M.D., Pham, Q.V., O’Sullivan, B., and Nguyen, H.D. (2025). Multi-Agent Collaboration Mechanisms: A Survey of LLMs. arXiv.
35. Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., and Yang, C. (2025). Scaling Large Language Model-based Multi-Agent Collaboration. arXiv.
36. Bee, A. (2025, April 22). Ciphey: Automatically Decrypt Encryptions Without Knowing the Key or Cipher, Decode Encodings, and Crack Hashes. Available online: https://github.com/bee-san/Ciphey.
37. He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. (2024). WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. arXiv.10.18653/v1/2024.acl-long.371 ↗
38. Dupont, D. (2025, April 22). GPT-4V-Act: AI Agent Using GPT-4V(ision) Capable of Using a Mouse/Keyboard to Interact with Web UI. Available online: https://github.com/ddupont808/GPT-4V-Act.
39. Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., Xi, H., Milner, K., Chen, B., Yin, M., Garg, S., and Krishnamurthy, P. (2024). NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security. arXiv.
40. Olson, E. (2025, April 22). Quipqiup: Cryptoquip and Cryptogram Solver. Available online: https://quipqiup.com.