An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture
Source abstract
Abstract
Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Smart Agriculture and AI · Robot Manipulation and Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute for Bio-Economy and Agri-Technology (IBO), Centre of Research and Technology-Hellas (CERTH), 6th km Charilaou-Thermi Rd, GR 57001 Thessaloniki, Greece
Department of Computer Science & Telecommunications, University of Thessaly, GR 35131 Lamia, Greece
Department of Agriculture, Forestry and Food Science (DISAFA), University of Turin, Largo Braccini 2, 10095 Grugliasco, Italy
Lincoln Institute for Agri-Food Technology (LIAT), University of Lincoln, Lincoln LN6 7TS, UK
References (78)
Show source-provided references
1. Oliveira, L.F.P., Moreira, A.P., and Silva, M.F. (2021). Advances in Agriculture Robotics: A State-of-the-Art Review and Challenges Ahead. Robotics, 10.10.3390/robotics10020052 ↗
2. Bechar, A. (2021). Agricultural Robotics for Precision Agriculture Tasks: Concepts and Principles. Innovation in Agricultural Robotics for Precision Agriculture: A Roadmap for Integrating Robots in Precision Agriculture, Springer International Publishing.10.1007/978-3-030-77036-5 ↗
3. Lampridi, M., Benos, L., Aidonis, D., Kateris, D., Tagarakis, A.C., Platis, I., Achillas, C., and Bochtis, D. (2021). The Cutting Edge on Advances in ICT Systems in Agriculture. Eng. Proc., 9.10.3390/engproc2021009046 ↗
4. A systematic literature review on applications of information and communication technologies and blockchain technologies for precision agriculture development10.1016/j.jclepro.2021.126763 ↗
5. Moysiadis, V., Tsolakis, N., Katikaridis, D., Sørensen, C.G., Pearson, S., and Bochtis, D. (2020). Mobile Robotics in Agricultural Operations: A Narrative Review on Planning Aspects. Appl. Sci., 10.10.3390/app10103453 ↗
6. Field Deployment of Robotic Systems for Agriculture in Light of Key Safety, Labor, Ethics and Legislation Issues10.1007/s43154-022-00074-9 ↗
7. Robotics and labour in agriculture. A context consideration10.1016/j.biosystemseng.2019.06.013 ↗
8. Agricultural robots for field operations. Part 2: Operations and systems10.1016/j.biosystemseng.2016.11.004 ↗
9. Human–robot interaction in agriculture: A survey and current challenges10.1016/j.biosystemseng.2018.12.005 ↗
10. Safety and ergonomics in human-robot interactive agricultural operations10.1016/j.biosystemseng.2020.09.009 ↗
11. Matheson, E., Minto, R., Zampieri, E.G.G., Faccio, M., and Rosati, G. (2019). Human–Robot Collaboration in Manufacturing Applications: A Review. Robotics, 8.10.3390/robotics8040100 ↗
12. A novel augmented reality-based interface for robot path planning10.1007/s12008-013-0191-2 ↗
13. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
14. Han, J., Campbell, N., Jokinen, K., and Wilcock, G. (2012, January 2–5). Investigating the use of Non-verbal Cues in Human-Robot Interaction with a Nao robot. Proceedings of the IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom), Kosice, Slovakia.10.1109/CogInfoCom.2012.6421937 ↗
15. Tran, D.-S., Ho, N.-H., Yang, H.-J., Baek, E.-T., Kim, S.-H., and Lee, G. (2020). Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network. Appl. Sci., 10.10.3390/app10020722 ↗
16. Varun, K.S., Puneeth, I., and Jacob, T.P. (2019, January 23–25). Virtual Mouse Implementation using Open CV. Proceedings of the 3rd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India.10.1109/ICOEI.2019.8862764 ↗
17. A case study of gesture-based games in enhancing the fine motor skills and recognition of children with autism10.1080/10494820.2018.1437048 ↗
18. Hand sign language recognition using multi-view hand skeleton10.1016/j.eswa.2020.113336 ↗
19. Autonomous Human-Vehicle Leader-Follower Control Using Deep-Learning-Driven Gesture Recognition10.3390/vehicles4010016 ↗
20. A Wireless Multi-Channel Capacitive Sensor System for Efficient Glove-Based Gesture Recognition with AI at the Edge
21. Dynamic Hand Gesture Recognition Based on Signals from Specialized Data Glove and Deep Learning Algorithms10.1109/TIM.2021.3077967 ↗
22. A multi-scale descriptor for real time RGB-D hand gesture recognition10.1016/j.patrec.2020.11.011 ↗
23. Jaramillo-Yánez, A., Benalcázar, M.E., and Mena-Maldonado, E. (2020). Real-Time Hand Gesture Recognition Using Surface Electromyography and Machine Learning: A Systematic Literature Review. Sensors, 20.10.3390/s20092467 ↗
24. Learning Data Correction for Myoelectric Hand Based on “Survival of the Fittest”10.34133/2021/9875814 ↗
25. Application Research on Optimization Algorithm of sEMG Gesture Recognition Based on Light CNN + LSTM Model10.34133/2021/9794610 ↗
26. Statistical Color Models with Application to Skin Detection10.1023/A:1013200319198 ↗
27. Real-Time Hand Gesture Recognition using Motion Tracking
28. SHREC 2021: Skeleton-based hand gesture recognition in the wild10.1016/j.cag.2021.07.007 ↗
29. Li, Y. (2012, January 22–24). Hand gesture recognition using Kinect. Proceedings of the IEEE International Conference on Computer Science and Automation Engineering, Beijing, China.
30. Real time hand detection in a complex background10.1016/j.engappai.2014.06.006 ↗
31. A survey of skin-color modeling and detection methods10.1016/j.patcog.2006.06.010 ↗
32. Real-time Motion-based Hand Gestures Recognition from Time-of-Flight Video10.1007/s11265-015-1090-5 ↗
33. Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review10.1016/j.compag.2020.105687 ↗
34. De Smedt, Q., Wannous, H., Vandeborre, J.-P., Guerry, J., Saux, B.L., and Filliat, D. (2017, January 23–24). 3D hand gesture recognition using a depth and skeletal dataset: SHREC’17 track. Proceedings of the Workshop on 3D Object Retrieval, Lyon, France.
35. Chen, Y., Luo, B., Chen, Y.-L., Liang, G., and Wu, X. (2015, January 6–9). A real-time dynamic hand gesture recognition system using kinect sensor. Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China.10.1109/ROBIO.2015.7419071 ↗
36. Xi, C., Chen, J., Zhao, C., Pei, Q., and Liu, L. (2018, January 25–27). Real-time Hand Tracking Using Kinect. Proceedings of the 2nd International Conference on Digital Signal Processing, Tokyo, Japan.10.1145/3193025.3193056 ↗
37. A Real-Time Hand Posture Recognition System Using Deep Neural Networks10.1145/2735952 ↗
38. Mujahid, A., Awan, M.J., Yasin, A., Mohammed, M.A., Damaševičius, R., Maskeliūnas, R., and Abdulkareem, K.H. (2021). Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model. Appl. Sci., 11.10.3390/app11094164 ↗
39. Agrawal, M., Ainapure, R., Agrawal, S., Bhosale, S., and Desai, S. (2020, January 30–31). Models for Hand Gesture Recognition using Deep Learning. Proceedings of the IEEE 5th International Conference on Computing Communication and Automation (ICCCA), Greater Noida, India.10.1109/ICCCA49541.2020.9250846 ↗
40. Niloy, E., Meghna, J., and Shahriar, M. (2021, January 8–9). Hand Gesture-Based Character Recognition Using OpenCV and Deep Learning. Proceedings of the International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI), Rajshahi, Bangladesh.10.1109/ACMI53878.2021.9528150 ↗
41. Devineau, G., Moutarde, F., Xi, W., and Yang, J. (2018, January 15–19). Deep Learning for Hand Gesture Recognition on Skeletal Data. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00025 ↗
42. Zengeler, N., Kopinski, T., and Handmann, U. (2019). Hand Gesture Recognition in Automotive Human-Machine Interaction Using Depth Cameras. Sensors, 19.10.3390/s19010059 ↗
43. RGB-D-based human motion recognition with deep learning: A survey10.1016/j.cviu.2018.04.007 ↗
44. Gesture recognition for human-robot collaboration: A review10.1016/j.ergon.2017.02.004 ↗
45. Benos, L., Tagarakis, A.C., Dolias, G., Berruto, R., Kateris, D., and Bochtis, D. (2021). Machine Learning in Agriculture: A Comprehensive Updated Review. Sensors, 21.10.3390/s21113758 ↗
46. Vasconez, J.P., Guevara, L., and Cheein, F.A. (2019, January 8–12). Social robot navigation based on HRI non-verbal communication: A case study on avocado harvesting. Proceedings of the ACM Symposium on Applied Computing, Limassol, Cyprus. Volume F147772.10.1145/3297280.3297569 ↗
47. Hurtado, J.P.V. (2020). Human-Robot Interaction Strategies in Agriculture, Universidad Técnica Federico Santa María.
48. Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network10.1155/2021/5256940 ↗
49. Tsolakis, N., Bechtsis, D., and Bochtis, D. (2019). AgROS: A Robot Operating System Based Emulation Tool for Agricultural Robotics. Agronomy, 9.10.3390/agronomy9070403 ↗
50. Benos, L., Tsaopoulos, D., and Bochtis, D. (2020). A Review on Ergonomics in Agriculture. Part II: Mechanized Operations. Appl. Sci., 10.10.3390/app10103484 ↗
51. Benos, L., Kokkotis, C., Tsatalas, T., Karampina, E., Tsaopoulos, D., and Bochtis, D. (2021). Biomechanical Effects on Lower Extremities in Human-Robot Collaborative Agricultural Tasks. Appl. Sci., 11.10.3390/app112411742 ↗
52. Tagarakis, A.C., Benos, L., Aivazidou, E., Anagnostis, A., Kateris, D., and Bochtis, D. (2021). Wearable Sensors for Identifying Activity Signatures in Human-Robot Collaborative Agricultural Environments. Eng. Proc., 9.10.3390/engproc2021009005 ↗
53. Anagnostis, A., Benos, L., Tsaopoulos, D., Tagarakis, A., Tsolakis, N., and Bochtis, D. (2021). Human activity recognition through recurrent neural networks for human-robot interaction in agriculture. Appl. Sci., 11.10.3390/app11052188 ↗
54. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M.G., and Lee, J. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv.
55. Bindhu, V., Tavares, J.M.R.S., and Du, K.-L. (2021, January 28–29). Hand Gesture Mapping Using MediaPipe Algorithm. Proceedings of the Third International Conference on Communication, Computing and Electronics Systems, Coimbatore, India.
56. Damindarov, R., Fam, C.A., Boby, R.A., Fahim, M., Klimchik, A., and Matsumaru, T. (2021, January 26–29). A depth camera-based system to enable touch-less interaction using hand gestures. Proceedings of the International Conference “Nonlinearity, Information and Robotics” (NIR), Innopolis, Russia.10.1109/NIR52917.2021.9666090 ↗
57. Boruah, B.J., Talukdar, A.K., and Sarma, K.K. (2021, January 15–17). Development of a Learning-aid tool using Hand Gesture Based Human Computer Interaction System. Proceedings of the Advanced Communication Technologies and Signal Processing (ACTS), Rourkela, India.10.1109/ACTS53447.2021.9708354 ↗
58. MediaPipe (2022, April 13). MediaPipe Hands. Available online: https://google.github.io/mediapipe/solutions/hands.html.
59. SMOTE: Synthetic Minority Over-sampling Technique10.1613/jair.953 ↗
60. Scikit-learn: Machine Learning in Python
61. A review on machine learning algorithms10.1007/978-981-15-6202-0_51 ↗
62. Singh, A., Thakur, N., and Sharma, A. (2016, January 16–18). A review of supervised machine learning algorithms. Proceedings of the 3rd International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India.
63. Mandal, J.K., and Bhattacharya, D. (2020). Supervised Classification Algorithms in Machine Learning: A Survey and Review. Emerging Technology in Modelling and Graphics, Springer Singapore.10.1007/978-981-13-7403-6 ↗
64. (2022, April 15). NVIDIA Jetson: The AI platform for autonomous machines. Available online: https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/.
65. (2021, December 13). ROS-Robot Operating System. Available online: https://www.ros.org/.
66. Hinas, A., Ragel, R., Roberts, J., and Gonzalez, F. (2020). A Framework for Multiple Ground Target Finding and Inspection Using a Multirotor UAS. Sensors, 20.10.3390/s20010272 ↗
67. Tagarakis, A.C., Filippou, E., Kalaitzidis, D., Benos, L., Busato, P., and Bochtis, D. (2022). Proposing UGV and UAV Systems for 3D Mapping of Orchard Environments. Sensors, 22.10.3390/s22041571 ↗
68. Grimstad, L., and From, P.J. (2017). The Thorvald II Agricultural Robotic System. Robotics, 6.10.3390/robotics6040024 ↗
69. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision (ECCV 2014), Zurich, Switzerland.10.1007/978-3-319-10584-0 ↗
70. Foote, T. (2013, January 22–23). tf: The transform library. Proceedings of the IEEE Conference on Technologies for Practical Robot Applications (TePRA), Woburn, MA, USA.10.1109/TePRA.2013.6556373 ↗
71. (2022, June 28). Navigation: Package Summary. Available online: https://www.opera.com/client/upgraded.
72. Zheng, K. (2021). ROS Navigation Tuning Guide. Robot Operating System (ROS), Springer.10.1007/978-3-030-75472-3_6 ↗
73. Kateris, D., Kalaitzidis, D., Moysiadis, V., Tagarakis, A.C., and Bochtis, D. (2021). Weed Mapping in Vineyards Using RGB-D Perception. Eng. Proc., 9.10.3390/engproc2021009030 ↗
74. Hershberger, D., Gossow, D., Faust, J., and William, W. (2022, April 15). RVIZ Package Summary. Available online: http://wiki.ros.org/rviz.
75. Do you feel safe with your robot? Factors influencing perceived safety in human-robot interaction based on subjective and objective measures10.1016/j.ijhcs.2021.102744 ↗
76. Marinoudi, V., Lampridi, M., Kateris, D., Pearson, S., Sørensen, C.G., and Bochtis, D. (2021). The Future of Agricultural Jobs in View of Robotization. Sustainability, 13.10.3390/su132112109 ↗
77. Cellular Neural Networks: A Survey10.1016/S1474-6670(17)51490-1 ↗
78. A CNN-based chip for robot locomotion control10.1109/TCSI.2005.852211 ↗