Building a Realistic Virtual Simulator for Unmanned Aerial Vehicle Teleoperation
Source abstract
Abstract
Unmanned Aerial Vehicles (UAVs) support humans in performing an increasingly varied number of tasks. UAVs need to be remotely operated by a human pilot in many cases. Therefore, pilots require repetitive training to master the UAV movements. Nevertheless, training with an actual UAV involves high costs and risks. Fortunately, simulators are alternatives to face these difficulties. However, existing simulators lack realism, do not present flight information intuitively, and sometimes do not allow natural interaction with the human operator. This work addresses these issues through a framework for building realistic virtual simulators for the human operation of UAVs. First, the UAV is modeled in detail to perform a dynamic simulation in this framework. Then, the information of the above simulation is utilized to manipulate the elements in a virtual 3D operation environment developed in Unity 3D. Therefore, the interaction with the human operator is introduced with a proposed teleoperation algorithm and an input device. Finally, a meta-heuristic optimization procedure provides realism to the simulation. In this procedure, the flight information obtained from an actual UAV is used to optimize the parameters of the teleoperation algorithm. The quadrotor is adopted as the study case to show the proposal’s effectiveness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotic Path Planning Algorithms · Teleoperation and Haptic Systems · Distributed Control Multi-Agent Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Tecnológico Nacional de Mexico/IT de Tlalnepantla, Research and Postgraduate Division, Tlalnepantla de Baz 54070, Mexico
Postgraduate Department, Instituto Politécnico Nacional, CIDETEC, Mexico City 07700, Mexico
References (60)
Show source-provided references
1. Abushahma, R.I.H., Ali, M.A.M., Rahman, N.A.A., and Al-Sanjary, O.I. (2019, January 8–9). Comparative Features of Unmanned Aerial Vehicle (UAV) for Border Protection of Libya: A Review. Proceedings of the 2019 IEEE 15th International Colloquium on Signal Processing Its Applications (CSPA), Penang, Malaysia.10.1109/CSPA.2019.8695991 ↗
2. Maimaitijiang, M., Sagan, V., Sidike, P., Daloye, A.M., Erkbol, H., and Fritschi, F.B. (2020). Crop Monitoring Using Satellite/UAV Data Fusion and Machine Learning. Remote Sens., 12.10.3390/rs12091357 ↗
3. Huang, J., Tian, G., Zhang, J., and Chen, Y. (2021). On Unmanned Aerial Vehicles Light Show Systems: Algorithms, Software and Hardware. Appl. Sci., 11.10.3390/app11167687 ↗
4. Dubins Path Planning of Multiple UAVs for Tracking Contaminant Cloud. 17th IFAC World Congress10.3182/20080706-5-KR-1001.00964 ↗
5. Vehicle Teleoperation Interfaces10.1023/A:1011295826834 ↗
6. Mansfield, K., Eveleigh, T., Holzer, T.H., and Sarkani, S. (2013, January 12–14). Unmanned aerial vehicle smart device ground control station cyber security threat model. Proceedings of the 2013 IEEE International Conference on Technologies for Homeland Security (HST), Waltham, MA, USA.10.1109/THS.2013.6699093 ↗
7. Unmanned Water-Powered Aerial Vehicles: Theory and Experiments10.1109/ACCESS.2019.2895335 ↗
8. Liaw, J.F., and Tsai, P.H. (2017, January 13–17). Target prediction to improve human errors in robot teleoperation system. Proceedings of the 2017 International Conference on Applied System Innovation (ICASI), Sapporo, Japan.10.1109/ICASI.2017.7988185 ↗
9. High-Level Teleoperation System for Aerial Exploration of Indoor Environments10.3389/frobt.2019.00095 ↗
10. Besada, J.A., Campaña, I., Bergesio, L., Bernardos, A.M., and de Miguel, G. (2019, January 11–15). Drone Flight Planning for Safe Urban Operations: UTM Requirements and Tools. Proceedings of the 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Kyoto, Japan.10.1109/PERCOMW.2019.8730856 ↗
11. High-Level Multiple-UAV Cinematography Tools for Covering Outdoor Events10.1109/TBC.2019.2892585 ↗
12. Jahre, M., Göhringer, D., and Millet, P. (2021). Challenges in the Realm of Embedded Real-Time Image Processing. Towards Ubiquitous Low-Power Image Processing Platforms, Springer International Publishing.10.1007/978-3-030-53532-2 ↗
13. Ashokkumar, C.R., and York, G.W.P. (2017, January 13–16). Data science for decision aiding UAV control. Proceedings of the 2017 International Conference on Unmanned Aircraft Systems (ICUAS), Miami, FL, USA.10.1109/ICUAS.2017.7991299 ↗
14. Zhang, R., Cao, S., Zhao, K., Yu, H., and Hu, Y. (2021). A Hybrid-Driven Optimization Framework for Fixed-Wing UAV Maneuvering Flight Planning. Electronics, 10.10.3390/electronics10192330 ↗
15. Ribeiro, R., Ramos, J., Safadinho, D., Reis, A., Rabadão, C., Barroso, J., and Pereira, A. (2021). Web AR Solution for UAV Pilot Training and Usability Testing. Sensors, 21.10.3390/s21041456 ↗
16. Stöcker, C., Bennett, R., Nex, F., Gerke, M., and Zevenbergen, J. (2017). Review of the Current State of UAV Regulations. Remote Sens., 9.10.3390/rs9050459 ↗
17. The Educational Road is Open
18. Applications of georeferenced high-resolution images obtained with unmanned aerial vehicles. Part I: Description of image acquisition and processing10.1007/s11119-014-9355-8 ↗
19. Hing, J.T., Sevcik, K.W., and Oh, P.Y. (2009, January 10–15). Improving unmanned aerial vehicle pilot training and operation for flying in cluttered environments. Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, St. Louis, MO, USA.10.1109/IROS.2009.5354080 ↗
20. Turk, I., Ozbek, E., Ekici, S., and Karakoc, T.H. (2021). A conceptual design of a solar powered UAV and assessment for continental climate flight conditions. Int. J. Green Energy, 1–11.10.1080/15435075.2021.1954008 ↗
21. Kinoshita, T., and Imado, F. (2006, January 18–21). The Application of an UAV Flight Simulator - The Development of a New Point Mass Model for an Aircraft. Proceedings of the 2006 SICE-ICASE International Joint Conference, Busan, Korea.10.1109/SICE.2006.314658 ↗
22. De Paolis, L.T., Bourdot, P., and Mongelli, A. (2017). Math Model of UAV Multi Rotor Prototype with Fixed Wing Aerodynamic Structure for a Flight Simulator. Augmented Reality, Virtual Reality, and Computer Graphics, Springer International Publishing.
23. Fernando, H.C.T.E., De Silva, A.T.A., De Zoysa, M.D.C., Dilshan, K.A.D.C., and Munasinghe, S.R. (2013, January 17–20). Modelling, simulation and implementation of a quadrotor UAV. Proceedings of the 2013 IEEE 8th International Conference on Industrial and Information Systems, Peradeniya, Sri Lanka.10.1109/ICIInfS.2013.6731982 ↗
24. Jin, G.D., Gu, L.X., and Lu, L.B. (2009, January 26–27). UAV Simulator-Based Simulation of Flight Control System. Proceedings of the 2009 International Workshop on Intelligent Systems and Applications, Hangzhou, China.10.1109/IWISA.2009.5073226 ↗
25. Siddiqui, K.T.A., Feil-Seifer, D., Jiang, T., Jose, S., Liu, S., and Louis, S. (2017, January 11–13). Development of a Swarm UAV Simulator Integrating Realistic Motion Control Models For Disaster Operations. Proceedings of the ASME 2017 Dynamic Systems and Control Conference, Tysons, VA, USA.
26. Unmanned air vehicle research simulator-prototyping and testing of control and navigation systems10.4028/www.scientific.net/SSP.198.266 ↗
27. Manaï, M., Desbiens, A., and Gagnon, E. (2012, January 13–16). Identification of a UAV and Design of a Hardware-in-the-Loop System for Nonlinear Control Purposes. Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, Minneapolis, MN, USA.
28. Lizarraga, M., Dobrokhodov, V., Elkaim, G., Curry, R., and Kaminer, I. (2009, January 6–9). Simulink Based Hardware-in-the-Loop Simulator for Rapid Prototyping of UAV Control Algorithms. Proceedings of the AIAA Infotech@Aerospace Conference, Seattle, WA, USA.10.2514/6.2009-1843 ↗
29. Design and Hardware-in-the-Loop Integration of a UAV Microavionics System in a Manned–Unmanned Joint Airspace Flight Network Simulator10.1007/s10846-008-9270-1 ↗
30. Lugo-Cárdenas, I., Salazar, S., and Lozano, R. (2016, January 7–10). The MAV3DSim hardware in the loop simulation platform for research and validation of UAV controllers. Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, USA.10.1109/ICUAS.2016.7502657 ↗
31. Meng, W., Hu, Y., Lin, J., Lin, F., and Teo, R. (2015, January 9–12). ROS+unity: An efficient high-fidelity 3D multi-UAV navigation and control simulator in GPS-denied environments. Proceedings of the IECON 2015—41st Annual Conference of the IEEE Industrial Electronics Society, Yokohama, Japan.10.1109/IECON.2015.7392488 ↗
32. Zhang, M., Qin, H., Lan, M., Lin, J., Wang, S., Liu, K., Lin, F., and Chen, B.M. (2015, January 9–12). A high fidelity simulator for a quadrotor UAV using ROS and Gazebo. Proceedings of the IECON 2015—41st Annual Conference of the IEEE Industrial Electronics Society, Yokohama, Japan.10.1109/IECON.2015.7392534 ↗
33. Baidya, S., Shaikh, Z., and Levorato, M. (November, January 28). FlyNetSim: An Open Source Synchronized UAV Network Simulator Based on Ns-3 and Ardupilot. Proceedings of the 21st ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, MSWIM ’18, Montreal, QC, Canada.
34. Ferlini, A., Wang, W., and Pau, G. (2019, January 19). Corner-3D: A RF Simulator for UAV Mobility in Smart Cities. Proceedings of the ACM SIGCOMM 2019 Workshop on Mobile AirGround Edge Computing, Systems, Networks, and Applications, MAGESys’19, Beijing, China.10.1145/3341568.3342108 ↗
35. Mcmanus, I., Greer, D., and Walker, R. (August, January 29). Uav avionics’ hardware in the loop’simulator. Proceedings of the 10th Australian International Aerospace Congress, Barton, Australia.
36. Valavanis, K.P., Beard, R., Oh, P., Ollero, A., Piegl, L.A., and Shim, H. (2009, January 8–10). Multi-UAV Simulator Utilizing X-Plane. Proceedings of the 2nd International Symposium on UAVs, Reno, NV, USA.
37. De Rango, F., Palmieri, N., Santamaria, A.F., and Potrino, G. (2017, January 9–12). A simulator for UAVs management in agriculture domain. Proceedings of the 2017 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS), Seattle, WA, USA.10.23919/SPECTS.2017.8046780 ↗
38. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016, January 11–14). A Benchmark and Simulator for UAV Tracking. Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands.10.1007/978-3-319-46478-7 ↗
39. Imado, F., Abe, S., and Kinoshita, T. (2006, January 18–21). The Development of Three-Dimensional Scale Model UAV Simulator. Proceedings of the 2006 SICE-ICASE International Joint Conference, Busan, Korea.10.1109/SICE.2006.314849 ↗
40. UTSim: A framework and simulator for UAV air traffic integration, control, and communication
41. Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2017, January 12–15). AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. Proceedings of the Field and Service Robotics, Zurich, Switzerland.10.1007/978-3-319-67361-5_40 ↗
42. Zhang, F., Lyu, X., Wang, Y., Gu, H., and Li, Z. (2017, January 9–13). Modeling and Flight Control Simulation of a Quadrotor Tailsitter VTOL UAV. Proceedings of the AIAA Modeling and Simulation Technologies Conference, Grapevine, TX, USA.10.2514/6.2017-1561 ↗
43. Spong, M.W., Hutchinson, S., and Vidyasagar, M. (2020). Robot Modeling and Control, John Wiley & Sons.
44. Wells, D.A. (1967). Schaum’s Outline of Lagrangian Dynamics, McGraw-Hill. Schaum’s Outlines.
45. Systems Analysis and Dynamic Simulation 110.1111/j.1365-2338.1979.tb02245.x ↗
46. A local observability analysis method for a time-varying nonlinear system and its application in the continuous self-calibration system10.1007/s11432-018-9725-1 ↗
47. Dynamic system-optimal traffic assignment using a state space model10.1016/0191-2615(93)90017-5 ↗
48. Fathoni, M.F., and Wuryandari, A.I. (2015, January 1–5). Comparison between Euler, Heun, Runge-Kutta and Adams-Bashforth-Moulton integration methods in the particle dynamic simulation. Proceedings of the 2015 4th International Conference on Interactive Digital Media (ICIDM), Bandung, Indonesia.10.1109/IDM.2015.7516314 ↗
49. Virtual reality10.1109/45.666641 ↗
50. Okita, A. (2019). Learning C# Programming with Unity 3D, AK Peters/CRC Press.10.1201/9780429810251 ↗
51. Deep neural network approach in robot tool dynamics identification for bilateral teleoperation10.1109/LRA.2020.2974445 ↗
52. A review of haptic feedback teleoperation systems for micromanipulation and microassembly10.1109/TASE.2013.2245122 ↗
53. Evaluating input device usability as a function of task difficulty in a tracking task10.1080/00140139.2014.988755 ↗
54. Multi-objective meta-heuristic optimization in intelligent control: A survey on the controller tuning problem10.1016/j.asoc.2020.106342 ↗
55. Self tuning fuzzy PID type load and frequency controller10.1016/S0196-8904(03)00149-3 ↗
56. Caraffini, F., Santucci, V., and Milani, A. (2020). Evolutionary Computation & Swarm Intelligence, MDPI.
57. Kenneth, V. (1999). Price. An introduction to differential evolution. New Ideas in Optimization, McGraw-Hill.
58. Differential Evolution: A review of more than two decades of research10.1016/j.engappai.2020.103479 ↗
59. Mezura-Montes, E., Velázquez-Reyes, J., and Coello Coello, C.A. (2006, January 8–12). A Comparative Study of Differential Evolution Variants for Global Optimization. Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, GECCO ’06, Seattle, WA, USA.10.1145/1143997.1144086 ↗
60. Ljung, L., and Glad, T. (1994). Modeling of Dynamic Systems, Prentice-Hall, Inc.