A Multi-Agent Model for Automatic Test Scheme Generation via Experience Interaction and 2D-Simulation Evaluation
Source abstract
Abstract
With the rapid development of maritime intelligent systems and equipment, it has become increasingly urgent to effectively test the intelligence level and collaborative capabilities of these systems and devices. Currently, maritime intelligent systems and equipment testing is primarily conducted manually, involving analyzing the requirements for testing, generating test plans, and evaluating performance item by item. However, this manual approach faces challenges such as time-consuming and labor-intensive scheme planning, and overly simplistic test scenarios. Therefore, we propose a multi-agent model to automatically generate test schemes via Experience Interaction and 2D-simulation evaluation (MAEI-2D). MAEI-2D is designed to enable the automatic generation and optimization of test schemes for maritime systems and equipment by integrating large-scale task understanding, multi-agent collaboration, and two-dimensional simulation-based evaluation. It includes three agents, which perform generation, simulation, and evaluation, respectively. To improve the effectiveness of derivation from test description, an LLM-driven reasoning mechanism is introduced through natural language prompts. Experimental results on test scheme generation for maritime intelligent equipment demonstrate the performance of MAEI-2D.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: AI-based Problem Solving and Planning · Human-Automation Interaction and Safety · Artificial Intelligence in Games
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Beihang University, Beijing 100191, China
School of Computer, Hangzhou Dianzi University, Hangzhou 310018, China
Dalian Scientific Test and Control Technology Institute, Dalian 234643, China
CSSC Navigation Technology Co., Ltd., Beijing 102600, China
CSSC Systems Engineering Research Institute, Beijing 100094, China
References (36)
Show source-provided references
1. Enoiu, E., Mehmood, M.N., Felderer, M., and Afzal, W. (2025). Automated Test Generation. Proceedings of International Conference on Fundamentals of Software Engineering, Springer.10.1007/978-3-031-87054-5_3 ↗
2. Alismail, A., and Lanquillon, C. (2025). A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows. Proceedings of International Conference on Human-Computer Interaction, Springer.10.1007/978-3-031-93418-6_9 ↗
3. LLMScenario: Large Language Model Driven Scenario Generation10.1109/TSMC.2024.3392930 ↗
4. Exploring the role of large language models in the scientific method: From hypothesis to discovery10.1038/s44387-025-00019-5 ↗
5. Wang, Z., Liu, K., Li, G., and Jin, Z. (2024). HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. Proceedings of the 39th International Conference on Automated Software Engineering, IEEE/ACM.10.1145/3691620.3695501 ↗
6. Hua, W.Y. (2024). War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv.
7. Goecks, V.G., and Waytowich, N. (2024). COA-GPT: Generative pre-trained transformers for accelerated course of action development in military operations. Proceedings of the International Conference on Military Communication and Information Systems, IEEE.10.1109/ICMCIS61231.2024.10540749 ↗
8. Cao, P., Men, T., Liu, W., Zhang, J., Li, X., Lin, X., Sui, D., Cao, Y., Liu, K., and Zhao, J. (2025). Large Language Models for Planning: A Comprehensive and Systematic Survey. arXiv.
9. Guan, L., Valmeekam, K., Sreedharan, S., and Kambhampati, S. (2023). Leveraging pre-trained large language models to constructand utilize world models for model-based task planning. Advances in Neural Information Processing Systems, NeurIPS.10.52202/075280-3459 ↗
10. Wong, L., Mao, J., Sharma, P., Siegel, Z.S., Feng, J., Korneev, N., Tenenbaum, J.B., and Andreas, J. (2023). Learning adaptive planning representations with natural language guidance. arXiv.
11. Zhang, L., Jansen, P., Zhang, T., Clark, P., Chris, C.B., and Tandon, N. (2024). PDDLEGO: Iterative Planning in Textual Environments. arXiv.10.18653/v1/2024.starsem-1.17 ↗
12. Kang, J., Laroche, R., Yuan, X., Trischler, A., Liu, X., and Fu, J. (2023). Think Before You Act: Decision Transformers with Working Memory. Proceedings of the 40th International Conference on Machine Learning, PMLR.
13. Wang, X., Li, L., and Zeng, D. (2023). LDM2: A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement. Findings of the Association for Computational Linguistics, Proceedings of the 2023 Conference on Empirical Methods in Natural Language, ACL.10.18653/v1/2023.findings-emnlp.309 ↗
14. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models10.1109/TPAMI.2024.3511593 ↗
15. Qiao, S., Fang, R., Zhang, N., Zhu, Y., Chen, X., Deng, S., Jiang, Y., Xie, P., Huang, F., and Chen, H. (2024). Agent planning with world knowledge model. Advances in Neural Information Processing Systems, NeurIPS.10.52202/079017-3646 ↗
16. Fu, Y., Anantha, R., and Cheng, J. (2024). CAMPHOR: Collaborative Agents for Multi-input Planning and High-Order Reasoning On Device. arXiv.10.18653/v1/2025.realm-1.3 ↗
17. Lyu, Y., Niu, Z., Xie, Z., Zhang, C., Xu, T., Wang, Y., and Chen, E. (2024). Retrieve-plan-generation: An iterative planning and answering framework for knowledge-intensive LLM generation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, ACL.10.18653/v1/2024.emnlp-main.270 ↗
18. Kim, M., Bursztyn, V., Koh, E., Guo, S., and Hwang, S. (2024). Rada: Retrieval-augmented web agent planning with llms. Findings of the Association for Computational Linguistics, ACL.10.18653/v1/2024.findings-acl.802 ↗
19. Zhang, C., Goh, D., Deik, X., Li, D., Zhang, H., and Liu, Y. (2024). Meta-task planning for language agents. arXiv.
20. Nair, L., Trase, I., and Kim, M. (2025). Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options. arXiv.
21. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., and Nyczyk, P. (2024). Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, AAAI.10.1609/aaai.v38i16.29720 ↗
22. Xu, S., Pang, L., Shen, H., Cheng, X., and Chua, T.S. (2024). Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks. Proceedings of the ACM Web Conference, ACM.10.1145/3589334.3645363 ↗
23. Roy, S., Sengupta, S., Bonadiman, D., Mansour, S., and Gupta, A. (2024). FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL.10.18653/v1/2024.naacl-long.29 ↗
24. Ma, C., Zhao, H., Zhang, J., He, J., and Kong, L. (2024). Non-myopic Generation of Language Model for Reasoning and Planning. Proceedings of International Conference on Learning Representations, ICLR.
25. O’Brien, S., and Mike, L. (2023). Contrastive decoding improves reasoning in large language models. arXiv.
26. On games and simulators as a platform for development of artificial intelligence for command and control10.1177/15485129221083278 ↗
27. Doumanas, D., Soularidis, A., and Kotis, K. (2026). Causal Reasoning and Large Language Models for Military Decision-Making: Rethinking the Command Structures in the Era of Generative AI. AI, 7.10.3390/ai7010014 ↗
28. Shrivastava, A., Hullman, J., and Lamparth, M. (2024). Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations. arXiv.
29. Petrovic, N., Lebioda, K., Zolfaghari, V., Schamschurko, A., Kirchner, S., Purschke, N., Pan, F., and Knoll, A. (2024). LLM-Driven Testing for Autonomous Driving Scenarios. Proceedings of the 2nd International Conference on Foundation and Large Language Models, IEEE.10.1109/FLLM63129.2024.10852505 ↗
30. SceGene: Bio-inspired traffic scenario generation for autonomous driving testing10.1109/TITS.2021.3134661 ↗
31. OmniTester: Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles10.1007/s42154-025-00364-w ↗
32. Liang, L., Tan, C., Deng, Y., Cai, Y., Chen, T.Y., and Zheng, X. (2025). AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systems. arXiv.
33. Hosseini, M.R., Janjua, N.K., Mohsin, A., and Rahman, S.U. (2025). Generating Test Cases with Large Language Model: Frameworks and Challenges. Proceedings of 2025 IEEE International Conference on E-Business Engineering, IEEE.10.1109/ICEBE68123.2025.00041 ↗
34. Abdullin, A., Derakhshanfar, P., and Panichella, A. (2025). Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation. Proceedings of IEEE Conference on Software Testing, Verification and Validation, IEEE.10.1109/ICST62969.2025.10989033 ↗
35. Gu, S., Zhang, Q., Li, K., Fang, C., Tian, F., Zhu, L., Zhou, J., and Chen, Z. (2025). TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration. arXiv.
36. Lenke, D., Ferrer, J., and Chicano, F. (2025). Influence of External Dependency Retrieval and Prompt Engineering in Test Case Generation Using LLMs. Proceedings of International Conference on Intelligent Data Engineering and Automated Learning, Springer.10.1007/978-3-032-10486-1_15 ↗