Maneuver Decision-Making through Automatic Curriculum Reinforcement Learning without Handcrafted Reward Functions
Source abstract
Abstract
Maneuver decision-making is essential for autonomous air combat. However, previous methods usually make decisions to aim at the target instead of hitting the target and use discrete action spaces instead of continuous action spaces. While these simplifications make maneuver decision-making easier, they also make maneuver decision-making more unrealistic. Meanwhile, previous studies usually rely on handcrafted reward functions, which are troublesome to design. Therefore, to solve these problems, we propose an automatic curriculum reinforcement learning method that enables agents to maneuver effectively in air combat from scratch. On the basis of curriculum reinforcement learning, maneuver decision-making is divided into a series of sub-tasks from easy to difficult. Thus, agents can gradually learn how to complete a series of sub-tasks, from easy to difficult without handcrafted reward functions. The ablation studies show that automatic curriculum learning is essential for reinforcement learning; namely, agents cannot make effective decisions without curriculum learning. Simulations show that, after training, agents are able to make effective decisions given different states, including tracking, attacking, and escaping, which are both rational and interpretable.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Guidance and Control Systems · Military Defense Systems Analysis · Adversarial Robustness in Machine Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Aeronautics Engineering College, Air Force Engineering University, Xi’an 710038, China
Air Defence and Antimissile College, Air Force Engineering University, Xi’an 710051, China
Air Force Xi’an Flying College, Xi’an 710300, China
Funding
Natural Science Foundation of Shaanxi Province
References (35)
Show source-provided references
1. Mohammadzadeh, A., Sabzalian, M.H., Zhang, C., Castillo, O., Sakthivel, R., and El-Sousy, F.F. (2022). Modern Adaptive Fuzzy Control Systems, Springer Nature.10.1007/978-3-031-17393-6 ↗
2. Mohammadazadeh, A., Sabzalian, M.H., Castillo, O., Sakthivel, R., El-Sousy, F.F., and Mobayen, S. (2022). Neural Networks and Learning Algorithms in MATLAB, Springer Nature.10.1007/978-3-031-14571-1 ↗
3. Application of deep reinforcement learning in maneuver planning of beyond-visual-range air combat10.1109/ACCESS.2021.3060426 ↗
4. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
5. Eloy, G., David, W.C., Dzung, T., and Meir, P. (2020). A differential game approach for beyond visual range tactics. arXiv.
6. Background interpolation for on-line situation of capture zone of air-to-air missiles
7. Dantas, J.P.A., Costa, A.N., Medeiros, F.L.L., Geraldo, D., and Marcos, R.O.A.M. (2022). Supervised Machine Learning for Effective Missile Launch Based on Beyond Visual Range Air Combat Simulations. arXiv.10.1109/WSC57314.2022.10015384 ↗
8. Fan, Z., Xu, Y., Kang, Y., and Luo, D. (2022). Air combat maneuver decision method based on A3C deep reinforcement learning. Machines, 10.10.3390/machines10111033 ↗
9. Mnih, V., Badia, A.P., and Mirza, M. (2016, January 19–24). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
10. Research on UCAV maneuvering decision method based on heuristic reinforcement learning
11. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization10.21629/JSEE.2018.01.09 ↗
12. Pope, A.P., Ide, J.S., Micovic, D., Diaz, H., and Rosenbluth, D. (2021). Hierarchical Reinforcement Learning for Air-to-Air Combat. arxiv.10.1109/ICUAS51884.2021.9476700 ↗
13. Alex, G., Marc, G.B., Jacob, M., Rémi, M., and Koray, K. (2017, January 6–11). Automated curriculum learning for neural networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
14. Teacher–Student Curriculum Learning10.1109/TNNLS.2019.2934906 ↗
15. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
16. William, H.G., Brandon, H., Nicholay, T., Phillip, W., Cayden, C., Manuela, V., and Ruslan, S. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arxiv.
17. Zhang, Y.Z., Abbeel, P., and Pinto, L. (2020, January 6–12). Automatic Curriculum Learning through Value Disagreement. Proceedings of the Advances in Neural Information Processing Systems, Virtual Conference.
18. Castells, T., Weinzaepfel, P., and Revaud, J. (2020, January 6–12). SuperLoss: A Generic Loss for Robust Curriculum Learning. Proceedings of the Advances in Neural Information Processing Systems, Virtual Conference.
19. Jiang, L., Meng, D.Y., Zhao, Q., Shan, S.G., and Hauptmann, A. (2016, January 12–17). Self-paced curriculum learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v29i1.9608 ↗
20. Stretcu, O., Platanios, E.A., Mitchell, T.M., and Póczos, B. (2021). Coarse-to-fine curriculum learning. arXiv.
21. Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. (2018, January 1–3). Intrinsic motivation and automatic curricula via asymmetric self-play. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
22. Rane, S. (2020). Learning with Curricula for Sparse-Reward Tasks in Deep Reinforcement Learning. [Master’s Thesis, Massachusetts Institute of Technology].
23. Pascal, K., Yang, H., Carlo, D., Joni, P., and Jan, P. (2022, January 17–23). Curriculum reinforcement learning via constrained optimal transport. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
24. Wu, J., and Yevgeniy, V. (2022, January 17–23). Robust deep reinforcement learning through bootstrapped opportunistic curriculum. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
25. Huang, P., Xu, M., Zhu, J., Shi, L., Fang, F., and Zhao, D. (2022, January 1–9). Curriculum reinforcement learning using optimal transport via gradual domain adaptation. Proceedings of the Advances in Neural Information Processing Systems, Virtual Conference.
26. Three-dimensional aircraft terrain-following via real-time optimal control
27. Air-to-air missile launchable area based on target escape maneuver estimation
28. Sutton, R.S., Mcallester, D.A., and Singh, S.P. (2000, January 1–2). Policy gradient methods for reinforcement learning with function approximation. Proceedings of the Advances in Neural Information Processing Systems, Breckenridge, CO, USA.
29. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 6–11). Trust region policy optimization. Proceedings of the International Conference on Machine Learning, Lille, France.
30. Schulman, J., Wolski, F., and Dhariwal, P. (2017). Proximal policy optimization algorithms. arXiv.
31. Adrià, P.B., Bilal, P., Steven, K., Pablo, S., Alex, V., Daniel, G., and Charles, B. (2020). Agent57: Outperforming the Atari human benchmark. arXiv.
32. Feryal, B., and Edward, H. (2023). Human-timescale adaptation in an open-ended task space. arXiv.
33. High-speed quadrupedal locomotion by imitation-relaxation reinforcement learning10.1038/s42256-022-00576-3 ↗
34. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
35. Grandmaster level in StarCraft II using multi-agent reinforcement learning10.1038/s41586-019-1724-z ↗