Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning
Source abstract
Abstract
Safety is the primary concern when it comes to air traffic. In-flight safety between Unmanned Aircraft Vehicles (UAVs) is ensured through pairwise separation minima, utilizing conflict detection and resolution methods. Existing methods mainly deal with pairwise conflicts, however, due to an expected increase in traffic density, encounters with more than two UAVs are likely to happen. In this paper, we model multi-UAV conflict resolution as a multiagent reinforcement learning problem. We implement an algorithm based on graph neural networks where cooperative agents can communicate to jointly generate resolution maneuvers. The model is evaluated in scenarios with 3 and 4 present agents. Results show that agents are able to successfully solve the multi-UAV conflicts through a cooperative strategy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Air Traffic Management and Optimization · Autonomous Vehicle Technology and Safety · Traffic control and management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Logistic and Aeronautics Group, Department of Telecommunications and System Engineering, Autonomous University of Barcelona, 08202 Sabadell, Spain
Funding
SESAR Joint Undertaking · Spanish National Project
References (48)
Show source-provided references
1. SESAR JU (2016). European Drones Outlook Study, SESAR JU. Single European Sky ATM Research.
2. Barrado, C., Boyero, M., Brucculeri, L., Ferrara, G., Hately, A., Hullah, P., Martin-Marrero, D., Pastor, E., Rushton, A.P., and Volkert, A. (2020). U-space concept of operations: A key enabler for opening airspace to emerging low-altitude operations. Aerospace, 7.10.3390/aerospace7030024 ↗
3. Prevot, T., Rios, J., Kopardekar, P., Robinson, J.E., Johnson, M., and Jung, J. (2016, January 13–17). UAS Traffic Management (UTM) Concept of Operations to Safely Enable Low Altitude Flight Operations. Proceedings of the 16th AIAA Aviation Technology, Integration, and Operations Conference, Washington, DC, USA.10.2514/6.2016-3292 ↗
4. Zhang, J. (2018, January 6–8). UOMS in China. Proceedings of the EU-China APP Drone Workshop, Shenzhen, China.
5. (2021, October 11). UTM—A Common Framework with Core Principles for Global Harmonization. Available online: https://www.icao.int/safety/UA/Documents/UTM%20Framework%20Edition%203.pdf.
6. (2021, October 11). NASA Conflict Management Model, Available online: https://www.nasa.gov/sites/default/files/atoms/files/2020-johnson-nasa-faa.pdf.
7. Test analysis of a scalable UAV conflict management framework10.1177/0954410019875241 ↗
8. Consiglio, M., Muñoz, C., Hagen, G., Narkawicz, A., and Balachandran, S. (2016, January 25–29). ICAROUS: Integrated configurable algorithms for reliable operations of unmanned systems. Proceedings of the 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC), Sacramento, CA, USA.10.1109/DASC.2016.7778033 ↗
9. Johnson, S.C., Petzen, A.N., and Tokotch, D.S. (2021, October 11). Johnson_DetectAndAvoid Aviation 2017 Handout, Available online: https://nari.arc.nasa.gov/sites/default/files/Johnson_DetectAndAvoid%20Aviation%202017%20Handout.pdf.
10. Manfredi, G., and Jestin, Y. (2018, January 12–15). Are You Clear About ” Well Clear ”?. Proceedings of the 2018 International Conference on Unmanned Aircraft Systems (ICUAS), Dallas, TX, USA.10.1109/ICUAS.2018.8453405 ↗
11. Cook, B., Arnett, T., and Cohen, K. (2017). A Fuzzy Logic Approach for Separation Assurance and Collision Avoidance for Unmanned Aerial Systems. Modern Fuzzy Control Systems and Its Applications, InTech.10.5772/68126 ↗
12. Consiglio, M., Duffy, B., Balachandran, S., Glaab, L., and Muñoz, C. (2019, January 17–21). Sense and avoid characterization of the independent configurable architecture for reliable operations of unmanned systems. Proceedings of the 13th USA/Europe Air Traffic Management Research and Development Seminar 2019, Vienna, Austria.
13. Unmanned aircraft systems in the national airspace system: A formal methods perspective10.1145/2984450.2984459 ↗
14. Well-clear recommendation for small unmanned aircraft systems based on unmitigated collision risk10.2514/1.D0091 ↗
15. McLain, T.W., and Duffield, M.O. (2017, January 9–13). A Well Clear Recommendation for Small UAS in High-Density, ADS-B-Enabled Airspace. Proceedings of the AIAA Information Systems-AIAA Infotech @ Aerospace, Grapevine, TX, USA.10.2514/6.2017-0908 ↗
16. Modi, H.C., Ishihara, A.K., Jung, J., Nikaido, B.E., D’souza, S.N., Hasseeb, H., and Johnson, M. (2016, January 25–30). Applying Required Navigation Performance Concept for Traffic Management of Small Unmanned Aircraft Systems. Proceedings of the 30th Congress of the International Council of the Aeronautical Sciences, Daejeon, Korea.
17. Jiang, J., Dun, C., Huang, T., and Lu, Z. (2018). Graph convolutional reinforcement learning. arXiv.
18. Sense and avoid for small unmanned aircraft systems: Research on methods and best practices10.1177/0954410019867802 ↗
19. A review of conflict detection and resolution modeling methods10.1109/6979.898217 ↗
20. Ribeiro, M., Ellerbroek, J., and Hoekstra, J. (2020). Review of conflict resolution methods for manned and unmanned aviation. Aerospace, 7.10.3390/aerospace7060079 ↗
21. Bertram, J., and Wei, P. (2020, January 6–10). Distributed computational guidance for high-density urban air mobility with cooperative and non-cooperative collision avoidance. Proceedings of the AIAA Scitech 2020 Forum, Orlando, FL, USA.10.2514/6.2020-1371 ↗
22. Scalable multiagent computational guidance with separation assurance for autonomous urban air mobility10.2514/1.G005000 ↗
23. Hu, J., Yang, X., Wang, W., Wei, P., Ying, L., and Liu, Y. (2020, January 15–19). UAS Conflict Resolution in Continuous Action Space Using Deep Reinforcement Learning. Proceedings of the AIAA Aviation 2020 Forum, Virtual Event.10.2514/6.2020-2909 ↗
24. Ribeiro, M., Ellerbroek, J., and Hoekstra, J. (2020, January 7–10). Determining Optimal Conflict Avoidance Manoeuvres At High Densities With Reinforcement Learning. Proceedings of the 10th SESAR Innovation Days, Virtual Event.
25. Isufaj, R., Aranega Sebastia, D., and Piera, M.A. (2021, January 20–24). Towards Conflict Resolution with Deep Multi-Agent Reinforcement Learning. Proceedings of the 14th USA/Europe Air Traffic Management Research and Development Seminar (ATM2021), New Orleans, LA, USA.
26. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv.
27. Pham, D.T., Tran, N.P., Alam, S., Duong, V., and Delahaye, D. (2019, January 7–21). A machine learning approach for conflict resolution in dense traffic scenarios with uncertainties. Proceedings of the ATM Seminar 2019, 13th USA/Europe ATM R&D Seminar, Vienna, Austria.
28. Brittain, M., Yang, X., and Wei, P. (2020). A deep multiagent reinforcement learning approach to autonomous separation assurance. arXiv.
29. Dalmau, R., and Allard, E. (2020, January 7–10). Air Traffic Control Using Message Passing Neural Networks and Multi-Agent Reinforcement Learning. Proceedings of the 10th SESAR Innovation Days, Virtual Event.
30. Manfredi, G., and Jestin, Y. (2016, January 25–29). An introduction to ACAS Xu and the challenges ahead. Proceedings of the 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC), Sacramento, CA, USA.10.1109/DASC.2016.7778055 ↗
31. Sutton, R.S., and Barto, A.G. (1998). Introduction to Reinforcement Learning, MIT Press.10.1109/TNN.1998.712192 ↗
32. Attention models in graphs: A survey10.1145/3363574 ↗
33. A survey and critique of multiagent deep reinforcement learning10.1007/s10458-019-09421-1 ↗
34. Dynamic programming and stochastic control processes10.1016/S0019-9958(58)80003-0 ↗
35. François-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., and Pineau, J. (2018). An introduction to deep reinforcement learning. arXiv.10.1561/9781680835397 ↗
36. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
37. Littman, M.L. (1994). Markov games as a framework for multiagent reinforcement learning. Machine Learning Proceedings 1994, Proceedings of the Eleventh International Conference, Rutgers University, New Brunswick, NJ, USA, 10–13 July 1994, Elsevier.10.1016/B978-1-55860-335-6.50027-1 ↗
38. A comprehensive survey on graph neural networks10.1109/TNNLS.2020.2978386 ↗
39. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv.
40. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
41. Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., and Dahl, G.E. (2017, January 6–11). Neural message passing for quantum chemistry. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
42. Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., and Lockhart, E. (2018). Relational deep reinforcement learning. arXiv.
43. Koca, T., Isufaj, R., and Piera, M.A. (2019, January 2–5). Strategies to Mitigate Tight Spatial Bounds Between Conflicts in Dense Traffic Situations. Proceedings of the 9th SESAR Innovation Days, Athens, Greece.
44. Hoekstra, J.M., and Ellerbroek, J. (2016, January 20–24). Bluesky ATC simulator project: An open data and open source approach. Proceedings of the 7th International Conference on Research in Air Transportation, FAA/Eurocontrol USA/Europe, Philadelphia, PA, USA.
45. Shi, K., Cai, K., Liu, Z., and Yu, L. (2020, January 11–15). A Distributed Conflict Detection and Resolution Method for Unmanned Aircraft Systems Operation in Integrated Airspace. Proceedings of the 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC), San Antonio, TX, USA.10.1109/DASC50938.2020.9256477 ↗
46. Mullins, M., Holman, M.W., Foerster, K., Kaabouch, N., and Semke, W. (2013, January 19–22). Dynamic Separation Thresholds for a Small Airborne Sense and Avoid System. Proceedings of the AIAA Infotech@Aerospace (I@A) Conference, Boston, MA, USA.10.2514/6.2013-5148 ↗
47. Improved Conflict Detection and Resolution for Service UAVs in Shared Airspace10.1109/TVT.2018.2889459 ↗
48. Johnson, S.C., Petzen, A.N., and Tokotch, D.S. (2017, January 5–9). Exploration of Detect-and-Avoid and Well-Clear Requirements for Small UAS Maneuvering in an Urban Environment. Proceedings of the 17th AIAA Aviation Technology, Integration, and Operations Conference, Denver, CO, USA.10.2514/6.2017-3074 ↗