Situational Awareness and Problems of Its Formation in the Tasks of UAV Behavior Control
Source abstract
Abstract
Situational awareness formation is one of the most critical elements in solving the problem of UAV behavior control. It aims to provide information support for UAV behavior control according to its objectives and tasks to be completed. We consider the UAV to be a type of controlled dynamic system. The article shows the place of UAVs in the hierarchy of dynamic systems. We introduce the concepts of UAV behavior and activity and formulate requirements for algorithms for controlling UAV behavior. We propose the concept of situational awareness as applied to the problem of behavior control of highly autonomous UAVs (HA-UAVs) and analyze the levels and types of this situational awareness. We show the specifics of situational awareness formation for UAVs and analyze its differences from situational awareness for manned aviation and remotely piloted UAVs. We propose the concept of situational awareness as applied to the problem of UAV behavior control and analyze the levels and types of this situational awareness. We highlight and discuss in more detail two crucial elements of situational awareness for HA-UAVs. The first of them is related to the analysis and prediction of the behavior of objects in the vicinity of the HA-UAV. The general considerations involved in solving this problem, including the problem of analyzing the group behavior of such objects, are discussed. As an illustrative example, the solution to the problem of tracking an aircraft maneuvering in the vicinity of a HA-UAV is given. The second element of situational awareness is related to the processing of visual information, which is one of the primary sources of situational awareness formation required for the operation of the HA-UAV control system. As an example here, we consider solving the problem of semantic segmentation of images processed when selecting a landing site for the HA-UAV in unfamiliar terrain. Both of these problems are solved using machine learning methods and tools. In the field of situational awareness for HA-UAVs, there are several problems that need to be solved. We formulate some of these problems and briefly describe them.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Technology and Human Factors in Education and Health · Advanced Data Processing Techniques · Aerospace, Electronics, Mathematical Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Flight Dynamics and Control, Moscow Aviation Institute (National Research University), Volokolamskoe Highway 4, 125080 Moscow, Russia
Funding
Ministry of Science and Higher Education of Russian Federation
References (100+)
Show source-provided references
1. Finn, A., and Scheding, S. (2010). Developments and Challenges for Autonomous Unmanned Vehicles, Springer.10.1007/978-3-642-10704-7 ↗
2. Valavanis, K.P. (2007). Advances in Unmanned Aerial Vehicles: State of the Art and the Road to Autonomy, Springer.10.1007/978-1-4020-6114-1 ↗
3. Martynyuk, A.A., and Martynyuk-Chernienko, Y.A. (2012). Uncertain Dynamic Systems: Stability and Motion Control, CRC Press.10.1201/b11314 ↗
4. Endsley, M.R., and Garland, D.J. (2000). Situation Awareness Analysis and Measurement, Lawrence Erlbaum Associates Inc.10.1201/b12461 ↗
5. Endsley, M.R., and Jones, D.G. (2004). Designing of Situation Awareness: An Approach to User-Centered Design, CRC Press. [2nd ed.].
6. Toward a theory of situation awareness in dynamic systems10.1518/001872095779049543 ↗
7. Situation awareness misconceptions and misunderstandings10.1177/1555343415572631 ↗
8. Situation awareness: Operationally necessary and scientifically grounded10.1007/s10111-015-0323-5 ↗
9. From here to autonomy: Lessons learned from human-automation research10.1177/0018720816681350 ↗
10. The limits of highly autonomous vehicles: An uncertain future10.1080/00140139.2019.1563330 ↗
11. Situation awareness in future autonomous vehicles: Beware of the unexpected10.1007/978-3-319-96071-5_32 ↗
12. The divergence of objective and subjective situation awareness: A meta-analysis10.1177/1555343419874248 ↗
13. Walker, G.H., Stanton, N.A., and Salmon, P.M. (2018). Vehicle Feedback and Driver Situation Awareness, CRC Press.10.1201/9781315578163 ↗
14. Distributed situation awareness in dynamic systems: Theoretical development and application of an ergonomics methodology10.1080/00140130600612762 ↗
15. Situation awareness measurement: A review of applicability for C4i environments10.1016/j.apergo.2005.02.001 ↗
16. What really is going on? Review of situation awareness models for individuals and teams10.1080/14639220701561775 ↗
17. Salmon, P.M., Stanton, N.A., Walker, G.H., and Jenking, D.P. (2009). Distributed Situation Awareness: Theory, Measurement and Application to Teamwork, MPG Books Group.
18. Measuring situation awareness in complex systems: Comparison of measures study10.1016/j.ergon.2008.10.010 ↗
19. Is situation awareness all in the mind?10.1080/14639220903009938 ↗
20. State-of-science: Situation awareness in individuals, teams and systems10.1080/00140139.2017.1278796 ↗
21. Salmon, P.M., Read, G.J.M., Walker, G.H., Lenne, M.G., and Stanton, N.A. (2019). Distributed Situation Awareness in Road Transport, CRC Press.10.4324/9781315609256 ↗
22. Some pitfalls in the promises of automated and autonomous vehicles10.1080/00140139.2018.1498136 ↗
23. Van der Laar, P., Tretmans, J., and Borth, M. (2013). Situation Awareness with Systems of Systems, Springer.10.1007/978-1-4614-6230-9 ↗
24. Gawron, V.J. (2019). Human Performance and Situation Awareness Measures, CRC Press. [3rd ed.].
25. Wise, J.A., Hopkin, V.D., and Garland, D.J. (2010). Handbook of Aviation Human Factors, CRC Press. [2nd ed.].
26. Angelov, P. (2012). Sense and Avoid in UAS: Research and Applications, John Wiley & Sons.10.1002/9781119964049 ↗
27. Situation awareness: A critical but ill-defined phenomenon10.1207/s15327108ijap0101_4 ↗
28. Automation situation awareness display for a flight envelope protection system10.2514/1.G000338 ↗
29. A review of situation awareness assessment approaches in aviation environment10.1109/JSYST.2019.2918283 ↗
30. Multi-sensor space debris tracking for situational awareness with labelled random finite sets10.1109/ACCESS.2019.2904545 ↗
31. Amos, B., Dinh, L., Cabi, S., Rothorl, T., Colmenarejo, S.G., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. (2018). Learning awareness models. arXiv.
32. Distribution of attention, situation awareness and workload in a passive air traffic control task: Implications for operational errors and automation10.2514/atcq.6.1.21 ↗
33. Use of real-time probes for measuring situation awareness10.1207/s15327108ijap1404_2 ↗
34. Development of an situation awareness software for autonomous unmanned aerial vehicles
35. A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques10.1007/s10846-018-0898-1 ↗
36. Relationship between UAVs and ambient objects with threat situational awareness through grid map-based ontology reasoning
37. Augmented reality tool for the situational awareness improvement of UAV operators10.3390/s17020297 ↗
38. Cuevas, H.M., and Aguiar, M. (2017). Assessing situation awareness in unmanned aircraft systems operations. Intern. J. Aviat. Aeronaut. Aerosp., 4.10.15394/ijaaa.2017.1176 ↗
39. Artificial situation awareness for increased autonomy of unmanned aerial systems in the terminal area10.1007/s10846-012-9738-x ↗
40. Towards artificial situation awareness by autonomous vehicles
41. Quantifying situation awareness for small unmanned aircraft10.1017/aer.2018.14 ↗
42. Enhanced situation awareness for unmanned aerial vehicle operating in terminal areas with circuit flight rules
43. Semantically enhanced UAVs to increase the aerial scene understanding10.1109/TSMC.2017.2757462 ↗
44. Bocaniala, C.D., and Sastry, V.V.S.S. (2010, January 6–13). On enhanced situational awareness models for unmanned aerial systems. Proceedings of the 2010 IEEE Aerospace Conference, Big Sky, MT, USA.10.1109/AERO.2010.5446781 ↗
45. Freedman, S.T., and Adams, J.A. (2007, January 7–10). The inherent components of unmanned vehicle situation awareness. Proceedings of the 2007 IEEE International Conference on Systems, Man and Cybernetics, Montreal, QC, Canada.10.1109/ICSMC.2007.4414129 ↗
46. Mission-driven autonomous perception and fusion based on UAV swarm10.1016/j.cja.2020.02.027 ↗
47. Hill, V.W., Thomas, R.W., and Larson, J.D. (2021). Autonomous situation awareness for UAS swarms. arXiv.10.1109/AERO50100.2021.9438461 ↗
48. Frische, F., and Lüdtke, A. (2013, January 25–28). SA-Tracer: A tool for assessment of UAV swarm operator SA during mission execution. Proceedings of the 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA), San Diego, CA, USA.10.1109/CogSIMA.2013.6523849 ↗
49. UAV-based situational awareness system using deep learning10.1109/ACCESS.2019.2938249 ↗
50. Carrio, A., Sampedro, C., Rodriguez-Ramos, A., and Campoy, P. (2017). A review od deep learning methods and applications for unmanned aerial vehicles. J. Sens., 1–13.10.1155/2017/3296874 ↗
51. Providing situational awareness in the control of unmanned vehicles10.1007/978-3-030-60577-3_14 ↗
52. Study on the situational awareness system of mine fire rescue using Faster Ross Girshick convolutional neural network10.1109/MIS.2019.2943850 ↗
53. Battlefield image situational awareness application based on deep learning10.1109/MIS.2019.2953685 ↗
54. A distributed event-driven architectural model based on situational awareness applied on internet of things10.1016/j.infsof.2019.04.001 ↗
55. A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems10.1177/0018720816634228 ↗
56. Modelling situation awareness relations in a multiagent system10.1007/s10489-015-0651-4 ↗
57. Van der Heijden, F., Duin, R.P., de Ridder, D., and Tax, D.M.J. (2004). Classification, Parameter Estimation and State Estimation, John Wiley & Sons.10.1002/0470090154 ↗
58. Hajiyev, C. (2010). State Estimation and Control for Low-Cost Unmanned Aerial Vehicles, Springer.
59. Isermann, R., and Münchhoh, M. (2011). Identification of Dynamic Systems: An Introduction with Applications, Springer.10.1007/978-3-540-78879-9 ↗
60. Ljung, L. (1999). System Identification: Theory for the User, Prentice-Hall, Inc.. [2nd ed.].
61. Wang, L., and Garnier, H. (2012). System Identification, Environment Modelling, and Control System Design, Springer.10.1007/978-0-85729-974-1 ↗
62. Nelles, O. (2020). Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models, Springer. [2nd ed.].10.1007/978-3-030-47439-3 ↗
63. Billings, S.A. (2013). Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-temporal Domains, John Wiley & Sons.10.1002/9781118535561 ↗
64. Klein, V., and Morelli, E.A. (2006). Aircraft System Identification: Theory and Practice, AIAA, Inc.10.2514/4.861505 ↗
65. Jategaonkar, R.V. (2006). Flight Vehicle System Identification: A Time Domain Methodology, AIAA, Inc.10.2514/4.866852 ↗
66. Tischler, M.B., and Remple, R.K. (2006). Aircraft and Rotorcraft System Identification: Engineering Methods with Flight-Test Examples, AIAA, Inc.
67. Talebi, H.A., Abdollahi, F., Patel, R.V., and Khorasani, K. (2004). Neural Network-Based State Estimation of Nonlinear Systems: Application to Fault Detection and Isolation, Springer.
68. Palade, V., and Bocaniala, C.D. (2006). Computational Intelligence in Fault Diagnosis, Springer.10.1007/978-1-84628-631-5 ↗
69. Blanke, M., Kinnaert, M., Lunze, J., and Staroswiecki, M. (2006). Diagnosis and Fault-Tolerant Control, Springer.
70. Patan, K. (2008). Artificial Neural Networks for the Modelling and Fault Diagnosis of Technical Processes, Springer.
71. Sobhani-Tehrani, E., and Khorasani, K. (2009). Fault Diagnosis of Nonlinear Systems Using a Hybrid Approach, Springer.10.1007/978-0-387-92907-1 ↗
72. Tiumentsev, Y.V., and Egorchev, M.V. (2019). Neural Network Modeling and Identification of Dynamical Systems, Elsevier.10.1007/978-3-030-30425-6_3 ↗
73. Katok, A., and Hasselblatt, B. (1995). Introduction to the Modern Theory of Dynamical Systems, Cambridge University Press.10.1017/CBO9780511809187 ↗
74. Ljung, L., and Glad, T. (1994). Modeling of Dynamic Systems, Prentice-Hall, Inc.
75. Khalil, H.K. (2002). Nonlinear Systems, Prentice Hall. [3rd ed.].
76. Brusov, V.S., and Tiumentsev, Y.V. (2016). Neural Network Based Modeling of Aircraft Motion, The MAI Publishing House. (In Russian).
77. Goodfellow, I., Bengio, Y., and Courville, A. (2017). Deep Learning, The MIT Press.
78. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.].
79. Dong, H., Ding, Z., and Zhang, S. (2020). Deep Reinforcement Learning: Fundamentals, Research and Applications, Springer.10.1007/978-981-15-4095-0 ↗
80. Kamalapurkar, R., Walters, P., Rosenfeld, J., and Dixon, W. (2018). Reinforcement Learning for Optimal Feedback Control: A Lyapunov-Based Approach, Springer.10.1007/978-3-319-78384-0 ↗
81. Ahrens, C.D., and Henson, R. (2018). Essentials of Meteorology Today: An Invitation to the Atmosphere, Cengage Learning. [8th ed.].
82. Ahrens, C.D., and Henson, R. (2019). Meteorology Today: An Introduction to Weather, Climate, and the Environment, Cengage Learning. [12th ed.].
83. Saha, K. (2008). The Earth’s Atmosphere: Its Physics and Dynamics, Springer.
84. Wells, N.C. (2012). The Atmosphere and Ocean: A Physical Introduction, John Wiley & Sons. [3rd ed.].
85. Njoku, E.G. (2014). Encyclopedia of Remote Sensing, Springer.10.1007/978-0-387-36699-9 ↗
86. Thenkabail, P.S. (2015). Remote Sensing Handbook. Vol. I: Remotely Sensed Data Characterization, Classification, and Accuracies, CRC Press.10.1201/b19294 ↗
87. Thenkabail, P.S. (2016). Remote Sensing Handbook. Vol. II, Land Resources Monitoring, Modeling, and Mapping with Remote Sensing, CRC Press.10.1201/b19322 ↗
88. Thenkabail, P.S. (2016). Remote Sensing Handbook. Vol. III, Remote Sensing of Water Resources, Disasters, and Urban Studies, CRC Press.10.1201/b19321 ↗
89. Adams, J.B., and Gillespie, A.R. (2006). Remote Sensing of Landscapes with Spectral Images: A Physical Modeling Approach, Cambridge University Press.10.1017/CBO9780511617195 ↗
90. Fraden, J. (2016). Handbook of Modern Sensors: Physics, Designs, and Applications, Springer. [5th ed.].10.1007/978-3-319-19303-8 ↗
91. Krasilshchikov, M.N., and Sebryakov, G.G. (2009). Modern Information Technologies Applying to Maneuverable Unmanned Vehicles Guidance, Navigation and Control Problems, Fizmatlit. (In Russian).
92. Liggins, M.E., Hall, D.L., and Llinas, J. (2009). Handbook of Multisensor Data Fusion: Theory and Practice, CRC Press. [2nd ed.].
93. Ellis, G. (2002). Observers in Control Systems, Academic Press.
94. (2021, December 01). The FlightGear Manual, September 2021. Available online: http://flightgear.sourceforge.net/getstart-en/getstart-en.html.
95. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
96. Phillips, D.J., Wheeler, T.A., and Kochenderfer, M.J. (2017, January 11–14). Generalizable intention prediction of human drivers at intersections. Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.10.1109/IVS.2017.7995948 ↗
97. A recurrent neural network solution for predicting driver intention at unsignalized intersections10.1109/LRA.2018.2805314 ↗
98. Learning to forget: Continual prediction with LSTM10.1162/089976600300015015 ↗
99. (2021, December 01). Semantic Drone Dataset. Institute of Computer Graphics and Vision, Graz University of Technology (Austria). Available online: http://dronedataset.icg.tugraz.at.
100. (2021, December 01). Aerial Semantic Segmentation Drone Dataset. Available online: https://www.kaggle.com/nunenuh/semantic-drone.