Enhancing Drone Navigation and Control: Gesture-Based Piloting, Obstacle Avoidance, and 3D Trajectory Mapping
Source abstract
Abstract
Autonomous drone navigation presents challenges for users unfamiliar with manual flight controls, increasing the risk of collisions. This research addresses this issue by developing a multifunctional drone control system that integrates hand gesture recognition, obstacle avoidance, and 3D mapping to improve accessibility and safety. The system utilizes Google’s MediaPipe Hands software library, which employs machine learning to track 21 key landmarks of the user’s hand, enabling gesture-based control of the drone. Each recognized gesture is mapped to a flight command, eliminating the need for a traditional controller. The obstacle avoidance system, utilizing the Flow Deck V2 and Multi-Ranger Deck, detects objects within a safety threshold and autonomously moves the drone by a predefined avoidance distance away to prevent collisions. A mapping system continuously logs the drone’s flight path and detects obstacles, enabling 3D visualization of drone’s trajectory after the drone landing. Also, an AI-Deck streams live video, enabling navigation beyond the user’s direct line of sight. Experimental validation with the Crazyflie drone demonstrates seamless integration of these systems, providing a beginner-friendly experience where users can fly drones safely without prior expertise. This research enhances human–drone interaction, making drone technology more accessible for education, training, and intuitive navigation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Robotic Path Planning Algorithms · UAV Applications and Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Sciences and Electrical Engineering, Marshall University, Huntington, WV 25755, USA
Funding
Defense Advanced Research Projects Agency (DARPA) · NASA West Virginia Space Grant Consortium, NASA Agreement · NASA Established Program to Stimulate Competitive Research
References (45)
Show source-provided references
1. Amicone, D., Cannas, A., Marci, A., and Tortora, G. (2021). A smart capsule equipped with artificial intelligence for autonomous delivery of medical material through drones. Appl. Sci., 11.10.3390/app11177976 ↗
2. Automating building damage reconnaissance to optimize drone mission planning for disaster response10.1061/(ASCE)CP.1943-5487.0001061 ↗
3. Drone-enabled multimodal platform for inspection of industrial components10.1109/ACCESS.2022.3167393 ↗
4. Nwaogu, J.M., Yang, Y., Chan, A.P., and Wang, X. (2024). Enhancing drone operator competency within the construction industry: Assessing training needs and roadmap for skill development. Buildings, 14.10.3390/buildings14041153 ↗
5. Tezza, D., Laesker, D., and Andujar, M. (2021, January 8–11). The learning experience of becoming a FPV drone pilot. Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, Boulder, CO, USA.10.1145/3434074.3447167 ↗
6. Lawrence, I.D., and Pavitra, A.R.R. (2024). Voice-controlled drones for smart city applications. Sustainable Innovation for Industry 6.0, IGI Global.
7. Shin, S.-Y., Kang, Y.-W., and Kim, Y.-G. (2019, January 11–13). Hand gesture-based wearable human-drone interface for intuitive movement control. Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.10.1109/ICCE.2019.8662106 ↗
8. Naseer, F., Ullah, G., Siddiqui, M.A., Khan, M.J., Hong, K.-S., and Naseer, N. (2022, January 4–7). Deep learning-based unmanned aerial vehicle control with hand gesture and computer vision. Proceedings of the 2022 13th Asian Control Conference (ASCC), Jeju Island, Republic of Korea.10.23919/ASCC56756.2022.9828347 ↗
9. Google (2025, May 04). Hand Landmarks Detection Guide. Available online: https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker.
10. Hayat, A., Li, C.H., Prakoso, N., Zheng, R., Wyawahare, A., and Wu, J. (2025). Gesture and Body Position Control for Lightweight Drones Using Remote Machine Learning Framework. Innovations in Electrical and Electronics Engineering, Proceedings of the ICEEE 2024, Marmaris, Turkey, 22–24 April 2024, Springer.10.1007/978-981-97-9112-5_2 ↗
11. Development of immersive virtual reality-based hand rehabilitation system using a gesture-controlled rhythm game with vibrotactile feedback: An fNIRS pilot study10.1109/TNSRE.2023.3312336 ↗
12. Manikanavar, A.R., and Shirol, S.B. (2022, January 25–26). Gesture controlled assistive device for deaf, dumb and blind people using Raspberry-Pi. Proceedings of the 2022 International Conference on Smart Technologies and Systems for Next Generation Computing (ICSTSN), Virtual.10.1109/ICSTSN53084.2022.9761314 ↗
13. Paterson, J., and Aldabbagh, A. (2021, January 11–13). Gesture-controlled robotic arm utilizing opencv. Proceedings of the 2021 3rd International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), Ankara, Turkey.10.1109/HORA52670.2021.9461389 ↗
14. Personalized Drone Interaction: Adaptive Hand Gesture Control with Facial Authentication10.32628/IJSRSET241146 ↗
15. Alanezi, M.A., Haruna, Z., Sha’aban, Y.A., Bouchekara, H.R., Nahas, M., and Shahriar, M.S. (2022). Obstacle avoidance-based autonomous navigation of a quadrotor system. Drones, 6.10.3390/drones6100288 ↗
16. Vision based drone obstacle avoidance by deep reinforcement learning10.3390/ai2030023 ↗
17. Ostovar, I. (2022). Nano-Drones: Enabling Indoor Collision Avoidance with a Miniaturized Multi-Zone Time of Flight Sensor. [Master’s Thesis, Politecnico di Torino].
18. OAST: Obstacle avoidance system for teleoperation of UAVs10.1109/THMS.2022.3142107 ↗
19. Bouwmeester, R.J., Paredes-Vallés, F., and De Croon, G.C. (June, January 29). Nanoflownet: Real-time dense optical flow on a nano quadcopter. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK.10.1109/ICRA48891.2023.10161258 ↗
20. Khoza, N., Owolawi, P., and Malele, V. (2024, January 7–8). Drone Gesture Control using OpenCV and Tello. Proceedings of the 2024 Conference on Information Communications Technology and Society (ICTAS), Durban, South Africa.10.1109/ICTAS59620.2024.10507111 ↗
21. Zhang, N., Nex, F., Vosselman, G., and Kerle, N. (2024). End-to-end nano-drone obstacle avoidance for indoor exploration. Drones, 8.10.3390/drones8020033 ↗
22. Telli, K., Kraa, O., Himeur, Y., Ouamane, A., Boumehraz, M., Atalla, S., and Mansoor, W. (2023). A comprehensive review of recent research trends on unmanned aerial vehicles (UAVs). Systems, 11.10.3390/systems11080400 ↗
23. Bitcraze (2025, April 18). Crazyflie 2.1. Available online: https://www.bitcraze.io/products/old-products/crazyflie-2-1/.
24. Bitcraze (2025, April 18). Crazyradio PA. Available online: https://www.bitcraze.io/products/crazyradio-pa/.
25. Bitcraze (2025, April 14). Flow Deck v2. Available online: https://store.bitcraze.io/products/flow-deck-v2.
26. Bitcraze (2025, April 14). Multi-Ranger Deck. Available online: https://store.bitcraze.io/products/multi-ranger-deck.
27. Bitcraze (2025, April 14). AI-Deck 1.1. Available online: https://store.bitcraze.io/products/ai-deck-1-1.
28. Yun, G., Kwak, H., and Kim, D.H. (2024). Single-Handed Gesture Recognition with RGB Camera for Drone Motion Control. Appl. Sci., 14.10.3390/app142210230 ↗
29. Lee, J.-W., and Yu, K.-H. (2023). Wearable drone controller: Machine learning-based hand gesture recognition and vibrotactile feedback. Sensors, 23.10.3390/s23052666 ↗
30. Natarajan, K., Nguyen, T.-H.D., and Mete, M. (2018, January 8–10). Hand gesture controlled drones: An open source library. Proceedings of the 2018 1st International Conference on Data Intelligence and Security (ICDIS), South Padre Island, TX, USA.10.1109/ICDIS.2018.00035 ↗
31. Begum, T., Haque, I., and Keselj, V. (2020, January 2–6). Deep learning models for gesture-controlled drone operation. Proceedings of the 2020 16th International Conference on Network and Service Management (CNSM), Virtual.10.23919/CNSM50824.2020.9269056 ↗
32. Reinforcement learning for shared autonomy drone landings10.1007/s10514-023-10143-3 ↗
33. Schwalb, J., Menon, V., Tenhundfeld, N., Weger, K., Mesmer, B., and Gholston, S. (2022, January 7–19). A study of drone-based AI for enhanced human-AI trust and informed decision making in human-AI interactive virtual environments. Proceedings of the 2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS), Orlando, FL, USA.10.1109/ICHMS56717.2022.9980625 ↗
34. Khaksar, S., Checker, L., Borazjan, B., and Murray, I. (2023). Design and Evaluation of an Alternative Control for a Quad-Rotor Drone Using Hand-Gesture Recognition. Sensors, 23.10.3390/s23125462 ↗
35. Moffatt, A., Platt, E., Mondragon, B., Kwok, A., Uryeu, D., and Bhandari, S. (2020, January 1–4). Obstacle detection and avoidance system for small UAVs using a LiDAR. Proceedings of the 2020 International Conference on Unmanned Aircraft Systems (ICUAS), Athens, Greece.10.1109/ICUAS48674.2020.9213897 ↗
36. Karam, S., Nex, F., Chidura, B.T., and Kerle, N. (2022). Microdrone-based indoor mapping with graph slam. Drones, 6.10.3390/drones6110352 ↗
37. Krul, S., Pantos, C., Frangulea, M., and Valente, J. (2021). Visual SLAM for indoor livestock and farming using a small drone with a monocular camera: A feasibility study. Drones, 5.10.3390/drones5020041 ↗
38. Sacoto-Martins, R., Madeira, J., Matos-Carvalho, J.P., Azevedo, F., and Campos, L.M. (2020, January 20–22). Multi-purpose low latency streaming using unmanned aerial vehicles. Proceedings of the 2020 12th International Symposium on Communication Systems, Networks and Digital Signal Processing (CSNDSP), Porto, Portugal.10.1109/CSNDSP49049.2020.9249562 ↗
39. Deep learning-based super-resolution reconstruction and marker detection for drone landing10.1109/ACCESS.2019.2915944 ↗
40. Real-time edge computing on multi-processes and multi-threading architectures for deep learning applications10.1016/j.micpro.2022.104554 ↗
41. Toward low-latency and ultra-reliable virtual reality10.1109/MNET.2018.1700268 ↗
42. Deep learning based hand gesture recognition and UAV flight controls10.1007/s11633-019-1194-7 ↗
43. Lyu, H. (2018, January 17–19). Detect and avoid system based on multi sensor fusion for UAV. Proceedings of the 2018 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Republic of Korea.
44. Habib, Y. (2024). Monocular SLAM Densification for 3D Mapping and Autonomous Drone Navigation. [Ph.D. Thesis, Ecole Nationale Supérieure Mines-Télécom Atlantique].
45. Gao, J., Lee, J., Zhou, Y., Hu, Y., Liu, C., and Zhu, P. (2024). SwarmCVT: Centroidal Voronoi Tessellation-Based Path Planning for Very-Large-Scale Robotics. arXiv.