Towards Seamless Human–Robot Interaction: Integrating Computer Vision for Tool Handover and Gesture-Based Control
Source abstract
Abstract
This paper presents the development of a robotic workstation that integrates a collaborative robot as an assistant, leveraging advanced computer vision techniques to enhance human–robot interaction. The system employs state-of-the-art computer vision models, YOLOv7 and YOLOv8, for precise tool detection and gesture recognition, enabling the robot to seamlessly interpret operator commands and hand over tools based on gestural cues. The primary objective is to facilitate intuitive, non-verbal control of the robot, improving collaboration between human operators and robots in dynamic work environments. The results show that this approach enhances the efficiency and reliability of human–robot cooperation, particularly in manufacturing settings, by streamlining tasks and boosting productivity. By integrating real-time computer vision into the robot’s decision-making process, the system demonstrates heightened adaptability and responsiveness, creating the way for more natural and effective human–robot collaboration in industrial contexts.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Robot Manipulation and Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Control and Information Systems, Faculty of Electrical Engineering and Information Technology, University of Zilina, 010 26 Zilina, Slovakia
Funding
APVV SK-IL-RD-23-0002: Advanced Localization Sensors and Techniques for Autonomous vehicles and Robots
References (59)
Show source-provided references
1. Digital innovation and the fourth industrial revolution: Epochal social changes?10.1007/s00146-017-0736-1 ↗
2. Castillo, J.F., Ortiz, J.H., Velásquez, M.F.D., and Saavedra, D.F. (2021). COBOTS in industry 4.0: Safe and efficient interaction. Collaborative and Humanoid Robots, IntechOpen.
3. Intelligent manufacturing systems: A review
4. Application, development and future opportunities of collaborative robots (cobots) in manufacturing: A literature review10.1080/10447318.2022.2041907 ↗
5. Nuzzi, C., Pasinetti, S., Lancini, M., Docchio, F., and Sansoni, G. (2018, January 16–18). Deep learning based machine vision: First steps towards a hand gesture recognition set up for collaborative robots. Proceedings of the 2018 Workshop on Metrology for Industry 4.0 and IoT, Brescia, Italy.10.1109/METROI4.2018.8439044 ↗
6. Michaelis, J.E., Siebert-Evenstone, A., Shaffer, D.W., and Mutlu, B. (2020, January 25–30). Collaborative or simply uncaged? Understanding human-cobot interactions in automation. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376547 ↗
7. Robots that interact with humans: A review of safety technologies and standards10.1108/IR-04-2017-0070 ↗
8. Recent progress on programming methods for industrial robots10.1016/j.rcim.2011.08.004 ↗
9. Deep learning for computer vision: A brief review10.1155/2018/7068349 ↗
10. Application of image processing in intelligent transport systems10.3182/20120523-3-CZ-3015.00012 ↗
11. Computer vision in healthcare applications10.1155/2018/5157020 ↗
12. Bubeníková, E., Franeková, M., and Holečko, P. (2013, January 23–26). Security increasing trends in intelligent transportation systems utilising modern image processing methods. Proceedings of the Activities of Transport Telematics: 13th International Conference on Transport Systems Telematics, TST 2013, Katowice-Ustroń, Poland. Selected Papers 13.10.1007/978-3-642-41647-7_43 ↗
13. Tirmizi, A., De Cat, B., Janssen, K., Pane, Y., Leconte, P., and Witters, M. (2019, January 23–24). User-friendly programming of flexible assembly applications with collaborative robots. Proceedings of the 2019 20th International Conference on Research and Education in Mechatronics (REM), Wels, Austria.10.1109/REM.2019.8744135 ↗
14. (2016). Robots and Robotic Devices—Collaborative Robots (Standard No. ISO/TS 15066).
15. Rao, R. (2024, February 29). Cobots vs. Robots: Understanding the Key Differences and Applications, [Online] 5. 7 2023. Available online: https://www.wevolver.com/article/cobots-vs-robots-understanding-the-key-differences-and-applications.
16. Arents, J., Abolins, V., Judvaitis, J., Vismanis, O., Oraby, A., and Ozols, K. (2021). Human–robot collaboration trends and safety aspects: A systematic review. J. Sens. Actuator Netw., 10.10.3390/jsan10030048 ↗
17. Safety assurance mechanisms of collaborative robotic systems in manufacturing10.1016/j.rcim.2020.102022 ↗
18. Wang, L., Liu, S., Liu, H., and Wang, X.V. (2020, January 1–4). Overview of human–robot collaboration in manufacturing. Proceedings of the 5th International Conference on the Industry 4.0 Model for Advanced Manufacturing: AMP 2020, Belgrade, Serbia.
19. Refining levels of collaboration to support the design and evaluation of human–robot interaction in the manufacturing industry10.1016/j.procir.2018.03.214 ↗
20. Deep learning-based multimodal control interface for human–robot collaboration10.1016/j.procir.2018.03.224 ↗
21. Bright, T., Adali, S., and Bright, G. (2022). Low-cost sensory glove for human–robot collaboration in advanced manufacturing systems. Robotics, 11.10.3390/robotics11030056 ↗
22. Sherwani, F., Asad, M.M., and Ibrahim, B.S.K.K. (2020, January 26–27). Collaborative robots and industrial revolution 4.0 (ir 4.0). Proceedings of the 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), Karachi, Pakistan.10.1109/ICETST49965.2020.9080724 ↗
23. Gesture recognition for human–robot collaboration: A review10.1016/j.ergon.2017.02.004 ↗
24. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques10.1109/TIM.2011.2161140 ↗
25. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand gesture recognition based on computer vision: A review of techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
26. Vision based gesture recognition system with single camera10.1109/ICOSP.2000.891796 ↗
27. Pinto, A.M., Costa, P., Moreira, A.P., Rocha, L.F., Veiga, G., and Moreira, E. (2023, January 26–27). Evaluation of depth sensors for robotic applications. Proceedings of the 2015 IEEE International Conference on Autonomous Robot Systems and Competitions, Tomar, Portugal.10.1109/ICARSC.2015.24 ↗
28. Electromagnetic Field & ToF Sensor Fusion for Advanced Perceptual Capability of Robots10.1109/LRA.2024.3386455 ↗
29. Saleem, Z., Gustafsson, F., Furey, E., McAfee, M., and Huq, S. (2024). A review of external sensors for human detection in a human robot collaborative environment. J. Intell. Manuf., 1–25.10.1007/s10845-024-02341-2 ↗
30. High level robot programming using body and hand gestures10.1016/j.procir.2016.09.020 ↗
31. Industrial robot control by means of gestures and voice commands in off-line and on-line mode10.3390/s20216358 ↗
32. Use of human gestures for controlling a mobile robot via adaptive cmac network and fuzzy logic controller10.1016/j.neucom.2017.12.016 ↗
33. Raj, T., Hanim Hashim, F., Baseri Huddin, A., Ibrahim, M.F., and Hussain, A. (2020). A survey on LiDAR scanning mechanisms. Electronics, 9.10.3390/electronics9050741 ↗
34. Innovative safety zoning for collaborative robots utilizing Kinect and LiDAR sensory approaches10.1016/j.procir.2022.02.180 ↗
35. Hu, J., Jiang, Z., Ding, X., Mu, T., and Hall, P. (2018, January 12–15). Vgpn: Voice-guided pointing robot navigation for humans. Proceedings of the 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia.10.1109/ROBIO.2018.8664854 ↗
36. Chamorro, S., Collier, J., and Grondin, F. (2021, January 25–27). Neural network based lidar gesture recognition for realtime robot teleoperation. Proceedings of the 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), New York, NY, USA.10.1109/SSRR53300.2021.9597855 ↗
37. Sensor glove based on novel inertial sensor fusion control algorithm for 3-D real-time hand gestures measurements10.1109/TIE.2019.2912765 ↗
38. Weber, P., Rueckert, E., Calandra, R., Peters, J., and Beckerle, P. (2016, January 26–31). A low-cost sensor glove with vibrotactile feedback and multiple finger joint and hand motion sensing for human–robot interaction. Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.10.1109/ROMAN.2016.7745096 ↗
39. Pan, S., Lv, H., Duan, H., Pang, G., Yi, K., and Yang, G. (2019, January 6–9). A sensor glove for the interaction with a nursing-care assistive robot. Proceedings of the 2019 IEEE International Conference on Industrial Cyber Physical Systems (ICPS), Taipei, Taiwan.10.1109/ICPHYS.2019.8780354 ↗
40. Côté-Allard, U., Gagnon-Turcotte, G., Laviolette, F., and Gosselin, B. (2019). A low-cost, wireless, 3-D-printed custom armband for sEMG hand gesture recognition. Sensors, 19.10.3390/s19122811 ↗
41. Loper, M.M., Koenig, N.P., Chernova, S.H., Jones, C.V., and Jenkins, O.C. (2009, January 9–13). Mobile human–robot teaming with environmental tolerance. Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, La Jolla, CA, USA.10.1145/1514095.1514124 ↗
42. Wolf, M.T., Assad, C., Vernacchia, M.T., Fromm, J., and Jethani, H.L. (2013, January 6–10). Gesture-based robot control with variable autonomy from the JPL BioSleeve. Proceedings of the 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany.10.1109/ICRA.2013.6630718 ↗
43. Hierarchical recurrent deep fusion using adaptive clip summarization for sign language translation10.1109/TIP.2019.2941267 ↗
44. Camgoz, N.C., Koller, O., Hadfield, S., and Bowden, R. (2020, January 14–19). Sign language transformers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.
45. Graph-based multimodal sequential embedding for sign language translation10.1109/TMM.2021.3117124 ↗
46. Kumar, S.S., Wangyal, T., Saboo, V., and Srinath, R. (2018, January 17–20). Time series neural networks for real time sign language translation. Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.
47. Herbert, O.M., Pérez-Granados, D., Ruiz, M.A.O., Cadena Martínez, R., Gutiérrez, C.A.G., and Antuñano, M.A.Z. (2024). Static and Dynamic Hand Gestures: A Review of Techniques of Virtual Reality Manipulation. Sensors, 24.10.3390/s24123760 ↗
48. Kwon, O.J., Kim, J., Lee, J., and Ullah, F. (2025). Vision-Based Gesture-Driven Drone Control in a Metaverse-Inspired 3D Simulation Environment. Drones, 9.10.3390/drones9020092 ↗
49. Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature Review10.1109/ACCESS.2024.3387269 ↗
50. Nákačka, J. (2023). Teoretický Prehľad v Oblasti Kolaboratívnej Robotiky. [Bachelor’s Thesis, Žilinská Univerzita].
51. Vavrecka, M., Sokovnin, N., Mejdrechova, M., and Sejnova, G. (2021, January 1–3). mygym: Modular toolkit for visuomotor robotic tasks. Proceedings of the 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), Virtual.10.1109/ICTAI52525.2021.00046 ↗
52. Kapitanov, A., Kvanchiani, K., Nagaev, A., Kraynov, R., and Makhliarchuk, A. (2024, January 3–8). HaGRID–HFurthermore, Gesture Recognition Image Dataset. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.
53. Szeliski, R. (2022). Computer Vision: Algorithms and Applications, Springer Nature.10.1007/978-3-030-34372-9 ↗
54. Hartley, R., and Zisserman, A. (2003). Multiple View Geometry in Computer Vision, Cambridge University Press.10.1017/CBO9780511811685 ↗
55. Brave new GES world: A systematic literature review of gestures and referents in gesture elicitation studies
56. Gesture-based human–robot interaction for human assistance in manufacturing10.1007/s00170-018-2788-x ↗
57. Mendes, N., Safeea, M., and Neto, P. (2018, January 18–20). Flexible programming and orchestration of collaborative robotic manufacturing systems. Proceedings of the 2018 IEEE 16th International Conference on Industrial Informatics (INDIN), Porto, Portugal.10.1109/INDIN.2018.8472058 ↗
58. Online recognition of incomplete gesture data to interface collaborative robots10.1109/TIE.2019.2891449 ↗
59. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗