journal article·2025-01-10Open access

Explainable AI-Enhanced Human Activity Recognition for Human–Robot Collaboration in Agriculture

Lefteris Benos, Dimitrios Tsaopoulos, Aristotelis C. Tagarakis, Dimitrios Kateris, Patrizia Busato, Dionysis Bochtis

Applied Sciences· Volume 15· Issue 2

Abstract

This study addresses a critical gap in human activity recognition (HAR) research by enhancing both the explainability and efficiency of activity classification in collaborative human–robot systems, particularly in agricultural environments. While traditional HAR models often prioritize improving overall classification accuracy, they typically lack transparency in how sensor data contribute to decision-making. To fill this gap, this study integrates explainable artificial intelligence, specifically SHapley Additive exPlanations (SHAP), thus enhancing the interpretability of the model. Data were collected from 20 participants who wore five inertial measurement units (IMUs) at various body positions while performing material handling tasks involving an unmanned ground vehicle in a field collaborative harvesting scenario. The results highlight the central role of torso-mounted sensors, particularly in the lumbar region, cervix, and chest, in capturing core movements, while wrist sensors provided useful complementary information, especially for load-related activities. The XGBoost-based model, selected mainly for allowing an in-depth analysis of feature contributions by considerably reducing the complexity of calculations, demonstrated strong performance in HAR. The findings indicate that future research should focus on enlarging the dataset, investigating the use of additional sensors and sensor placements, and performing real-world trials to enhance the model’s generalizability and adaptability for practical agricultural applications.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Muscle activation and electromyography studies · Context-Aware Activity Recognition Systems · Occupational Health and Safety Research

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Institute for Bio-Economy and Agri-Technology (IBO), Centre of Research and Technology-Hellas (CERTH), 6th km Charilaou-Thermi Rd., 57001 Thessaloniki, Greece

Interuniversity Department of Regional and Urban Studies and Planning (DIST), Polytechnic of Turin, Viale Mattioli 39, 10125 Torino, Italy

farmB Digital Agriculture S.A., 17th November 79, 55534 Thessaloniki, Greece

References (83)

Show source-provided references
1. Agricultural robots for field operations: Concepts and components10.1016/j.biosystemseng.2016.06.014 ↗
2. Agricultural robots for field operations. Part 2: Operations and systems10.1016/j.biosystemseng.2016.11.004 ↗
3. Transitioning towards human–robot synergy in agriculture: A systems thinking perspective10.1002/sres.2887 ↗
4. Collaboration of human pickers and crop-transporting robots during harvesting—Part I: Model and simulator development10.1016/j.compag.2020.105324 ↗
5. Collaboration of human pickers and crop-transporting robots during harvesting—Part II: Simulator evaluation and robot-scheduling case-study10.1016/j.compag.2020.105323 ↗
6. Benos, L., Moysiadis, V., Kateris, D., Tagarakis, A.C., Busato, P., Pearson, S., and Bochtis, D. (2023). Human-Robot Interaction in Agriculture: A Systematic Review. Sensors, 23.10.3390/s23156776 ↗
7. Human–Robot Collaboration in Modern Agriculture: A Review of the Current Research Landscape10.1002/aisy.202300823 ↗
8. Safety and ergonomics in human-robot interactive agricultural operations10.1016/j.biosystemseng.2020.09.009 ↗
9. Benefits of Human—Robot Interaction [TC Spotlight]10.1109/MRA.2011.943237 ↗
10. Marinoudi, V., Benos, L., Villa, C.C., Lampridi, M., Kateris, D., Berruto, R., Pearson, S., Sørensen, C.G., and Bochtis, D. (2024). Adapting to the Agricultural Labor Market Shaped by Robotization. Sustainability, 16.10.3390/su16167061 ↗
11. Recent trends in human activity recognition—A comparative study10.1016/j.cogsys.2022.10.003 ↗
12. Huang, Y., Zhou, Y., Zhao, H., Riedel, T., and Beigl, M. (July, January 30). A Survey on Wearable Human Activity Recognition: Innovative Pipeline Development for Enhanced Research and Practice. Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN), Yokohama, Japan.10.1109/IJCNN60899.2024.10650060 ↗
13. A survey of multi-agent Human–Robot Interaction systems10.1016/j.robot.2022.104335 ↗
14. Human–robot collaboration systems in agricultural tasks: A review and roadmap10.1016/j.compag.2022.107541 ↗
15. Do you feel safe with your robot? Factors influencing perceived safety in human-robot interaction based on subjective and objective measures10.1016/j.ijhcs.2021.102744 ↗
16. Field Deployment of Robotic Systems for Agriculture in Light of Key Safety, Labor, Ethics and Legislation Issues10.1007/s43154-022-00074-9 ↗
17. Moysiadis, V., Katikaridis, D., Benos, L., Busato, P., Anagnostis, A., Kateris, D., Pearson, S., and Bochtis, D. (2022). An Integrated Real-Time Hand Gesture Recognition Framework for Human-Robot Interaction in Agriculture. Appl. Sci., 12.10.3390/app12168160 ↗
18. Human–Robot Interaction through Dynamic Movement Recognition for Agricultural Environments10.3390/agriengineering6030146 ↗
19. A novel end-to-end vision-based architecture for agricultural human–robot collaboration in fruit picking operations10.1016/j.robot.2023.104567 ↗
20. Gokul, S., Dhiksith, R., Sundaresh, S.A., and Gopinath, M. (2019, January 15–16). Gesture Controlled Wireless Agricultural Weeding Robot. Proceedings of the 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS), Coimbatore, India.10.1109/ICACCS.2019.8728429 ↗
21. Patil, P.A., Jagyasi, B.G., Raval, J., Warke, N., and Vaidya, P.P. (2015, January 6–10). Design and development of wearable sensor textile for precision agriculture. Proceedings of the 2015 7th International Conference on Communication Systems and Networks, COMSNETS 2015–Proceedings, Bangalore, India.10.1109/COMSNETS.2015.7098714 ↗
22. Sharma, S., Raval, J., and Jagyasi, B. (2013, January 20–23). Mobile sensing for agriculture activities detection. Proceedings of the 3rd IEEE Global Humanitarian Technology Conference, GHTC, San Jose, CA, USA.10.1109/GHTC.2013.6713707 ↗
23. Sharma, S., Raval, J., and Jagyasi, B. (2014, January 11–13). Neural network based agriculture activity detection using mobile accelerometer sensors. Proceedings of the 11th IEEE India Conference: Emerging Trends and Innovation in Technology, INDICON, Pune, India.10.1109/INDICON.2014.7030539 ↗
24. Sharma, S., Jagyasi, B., Raval, J., and Patil, P. (2015, January 23–27). AgriAcT: Agricultural Activity Training using multimedia and wearable sensing. Proceedings of the 2015 IEEE International Conference on Pervasive Computing and Communication Workshops, PerCom Workshops, St. Louis, MO, USA.10.1109/PERCOMW.2015.7134078 ↗
25. Worker safety in agriculture 4.0: A new approach for mapping operator’s vibration risk through Machine Learning activity recognition10.1016/j.compag.2021.106637 ↗
26. Anagnostis, A., Benos, L., Tsaopoulos, D., Tagarakis, A., Tsolakis, N., and Bochtis, D. (2021). Human activity recognition through recurrent neural networks for human-robot interaction in agriculture. Appl. Sci., 11.10.3390/app11052188 ↗
27. Benos, L., Tsaopoulos, D., Tagarakis, A.C., Kateris, D., and Bochtis, D. (2024). Optimal Sensor Placement and Multimodal Fusion for Human Activity Recognition in Agricultural Tasks. Appl. Sci., 14.10.3390/app14188520 ↗
28. A Hybrid Attention-Based Deep Neural Network for Simultaneous Multi-Sensor Pruning and Human Activity Recognition10.1109/JIOT.2022.3196170 ↗
29. Energy-Efficient and Interpretable Multisensor Human Activity Recognition via Deep Fused Lasso Net10.1109/TETCI.2024.3430008 ↗
30. Artificial intelligence and explanation: How, why, and when to explain black boxes10.1016/j.ejrad.2024.111393 ↗
31. Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey10.1109/JIOT.2023.3287678 ↗
32. Borella, E., Çakmakçi, U.B., Gottardis, E., Buratto, A., Marchioro, T., and Badia, L. (2024, January 12–14). Effective Sensor Selection for Human Activity Recognition via Shapley Value. Proceedings of the 2024 IEEE International Workshop on Metrology for Living Environment (MetroLivEnv), Chania, Greece.10.1109/MetroLivEnv60384.2024.10615860 ↗
33. Arul, V., Karthikeyan, P., and Ramanujam, E. (2024, January 21–23). Revealing the Importance of Local and Global Interpretability in Smartphone Based Human Activity Recognition. Proceedings of the 2024 IEEE Students Conference on Engineering and Systems (SCES), Prayagraj, India.10.1109/SCES61914.2024.10652326 ↗
34. Junaid, M., Ali, S., Eid, F., El-Sappagh, S., and Abuhmed, T. (2023). Explainable machine learning models based on multimodal time-series data for the early detection of Parkinson’s disease. Comput. Methods Programs Biomed., 234.10.1016/j.cmpb.2023.107495 ↗
35. Open Datasets—iBO (2024, January 07). Explainable AI-Enhanced Human Activity Recognition for Human–Robot Collaboration in Agriculture. Available online: https://ibo.certh.gr/open-datasets/.
36. Pedretti, A., Gaya, A.C.A., Mello, J.B., and Gaya, A.R. (2024). Effect of the PROFIT-BR exercise program on physical fitness of children: A protocol study. BMC Sports Sci. Med. Rehabil., 16.10.1186/s13102-024-00990-7 ↗
37. Li, T., Kapilevich, L.V., and Chen, J. (2024). Effects of Mild Fatigue on Biomechanics of Single Leg Landing in Young Male Volleyball Players. Sensors, 24.10.3390/s24216811 ↗
38. The effects of lifting speed on the peak external forward bending, lateral bending, and twisting spine moments10.1080/001401399185838 ↗
39. Tagarakis, A.C., Benos, L., Aivazidou, E., Anagnostis, A., Kateris, D., and Bochtis, D. (2021). Wearable Sensors for Identifying Activity Signatures in Human-Robot Collaborative Agricultural Environments. Eng. Proc., 9.10.3390/engproc2021009005 ↗
40. Slattery, P., Cofré Lizama, L.E., Wheat, J., Gastin, P., Dascombe, B., and Middleton, K. (2024). The Agreement between Wearable Sensors and Force Plates for the Analysis of Stride Time Variability. Sensors, 24.10.3390/s24113378 ↗
41. Mascia, G., De Lazzari, B., and Camomilla, V. (2023). Machine learning aided jump height estimate democratization through smartphone measures. Front. Sports Act. Living, 5.10.3389/fspor.2023.1112739 ↗
42. A novel method for accurate division of the gait cycle into seven phases using shank angular velocity10.1016/j.gaitpost.2024.04.006 ↗
43. (2024, January 03). Blue Trident IMU | Inertial Sensor by Vicon | Biomechanic Tracking. Available online: https://www.vicon.com/hardware/blue-trident/.
44. (2024, August 19). VICON Capture.U 1.4: IOS app for Blue Trident Sensors. Available online: https://www.vicon.com/software/captureu/.
45. So many ways for assessing outliers: What really works and does it matter?10.1016/j.jbusres.2021.03.066 ↗
46. El-Adawi, E., Essa, E., Handosa, M., and Elmougy, S. (2024). Wireless body area sensor networks based human activity recognition using deep learning. Sci. Rep., 14.10.1038/s41598-024-53069-1 ↗
47. Sklearn.preprocessing (2021, January 20). StandardScaler—Scikit-Learn 0.24.1 Documentation. Available online: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html.
48. Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23.10.3390/e23010018 ↗
49. Chen, T., and Guestrin, C. (2016, January 25–29). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Barcelona, Spain.10.1145/2939672.2939785 ↗
50. Wu, X., Ma, L., Wei, P., Shan, Y., Chan, P., Wang, K., and Zhao, G. (2024). Wearable sensor devices can automatically identify the ON-OFF status of patients with Parkinson’s disease through an interpretable machine learning model. Front. Neurol., 15.10.3389/fneur.2024.1387477 ↗
51. Liu, Z., Zhang, Q., Zheng, H., Chen, S., and Gong, Y. (2024, January 18–20). A Comparative Study of Machine Learning Approaches for Diabetes Risk Prediction: Insights from SHAP and Feature Importance. Proceedings of the 2024 5th International Conference on Machine Learning and Computer Application (ICMLCA), Hangzhou, China.10.1109/ICMLCA63499.2024.10754491 ↗
52. Data-driven interpretable ensemble learning methods for the prediction of wind turbine power incorporating SHAP analysis10.1016/j.eswa.2023.121464 ↗
53. An interpretable multi-stage forecasting framework for energy consumption and CO2 emissions for the transportation sector10.1016/j.energy.2023.129499 ↗
54. Li, X., Deng, H., Yu, X., and Yu, Y. (2024). Cooperative Impact of the Digital Sector, Eco-Friendly Policies, and Sophisticated Economic Development: A Study Drawing from China’s Practices. Sustainability, 16.10.3390/su162310525 ↗
55. Felix, J., Alexandre, M., and Lima, G.T. (2024). Applying Machine Learning Algorithms to Predict the Size of the Informal Economy. Comput. Econ., 1–21.10.1007/s10614-024-10593-6 ↗
56. Retzepis, N.-O., Avloniti, A., Kokkotis, C., Protopapa, M., Stampoulis, T., Gkachtsou, A., Pantazis, D., Balampanos, D., Smilios, I., and Chatzinikolaou, A. (2024). Identifying Key Factors for Predicting the Age at Peak Height Velocity in Preadolescent Team Sports Athletes Using Explainable Machine Learning. Sports, 12.10.3390/sports12110287 ↗
57. Tatsuro, I. (1983). For a proof of unique existence. Game Theory for Economic Analysis, Academic Press.
58. Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence10.1016/j.inffus.2023.101805 ↗
59. Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence10.1007/s12559-023-10179-8 ↗
60. Explainable artificial intelligence: A survey of needs, techniques, applications, and future direction10.1016/j.neucom.2024.128111 ↗
61. From local explanations to global understanding with explainable AI for trees10.1038/s42256-019-0138-9 ↗
62. (2024, November 01). Scikit-Learn, Machine Learning in Python. Available online: https://scikit-learn.org/stable/.
63. Emmerzaal, J., Filtjens, B., Vets, N., Vanrumste, B., Smeets, A., De Groef, A., and De Baets, L. (2024). A data-driven approach to detect upper limb functional use during daily life in breast cancer survivors using wrist-worn sensors. Sci. Rep., 14.10.1038/s41598-024-67497-6 ↗
64. Carnegie, D.R., Hirsch, S.M., Beach, T.A., and Howarth, S.J. (2024). Restricting lumbar spine flexion redistributes and changes total mechanical energy expenditure during lifting. J. Biomech., 168.10.1016/j.jbiomech.2024.112132 ↗
65. Physiological impact of powered back-support exoskeletons in construction: Analyzing muscle fatigue, metabolic cost, ergonomic risks, and stability10.1016/j.autcon.2024.105742 ↗
66. Benos, L., Kokkotis, C., Tsatalas, T., Karampina, E., Tsaopoulos, D., and Bochtis, D. (2021). Biomechanical Effects on Lower Extremities in Human-Robot Collaborative Agricultural Tasks. Appl. Sci., 11.10.3390/app112411742 ↗
67. Load Carriage Modes and Limb Crossing Patterns Altered Gait during Obstacle Negotiation10.1080/00222895.2021.2017837 ↗
68. He, Z., Sun, Y., and Zhang, Z. (2024). Human Activity Recognition Based on Deep Learning Regardless of Sensor Orientation. Appl. Sci., 14.10.3390/app14093637 ↗
69. Advanced IoT-Based Human Activity Recognition and Localization Using Deep Polynomial Neural Network10.1109/ACCESS.2024.3420752 ↗
70. Human Activity Recognition (HAR) Using Deep Learning: Review, Methodologies, Progress and Future Research Directions10.1007/s11831-023-09986-x ↗
71. Design and Evaluation of an Interactive Exercise Coaching System for Older Adults: Lessons Learned10.1109/JBHI.2015.2391671 ↗
72. Camp, N., Lewis, M., Hunter, K., Johnston, J., Zecca, M., Di Nuovo, A., and Magistro, D. (2021). Technology Used to Recognize Activities of Daily Living in Community-Dwelling Older Adults. Int. J. Environ. Res. Public Health, 18.10.3390/ijerph18010163 ↗
73. Surface Electromyography and Artificial Intelligence for Human Activity Recognition—A Systematic Review on Methods, Emerging Trends Applications, Challenges, and Future Implementation10.1109/ACCESS.2023.3316509 ↗
74. Intuitive Human-Robot-Environment Interaction with EMG Signals: A Review10.1109/JAS.2024.124329 ↗
75. Evaluation of Biomechanical and Mental Workload During Human–Robot Collaborative Pollination Task10.1177/00187208241254696 ↗
76. Mental Workload and Human-Robot Interaction in Collaborative Tasks: A Scoping Review10.1080/10447318.2023.2254639 ↗
77. Grimstad, L., and From, P.J. (2017). The Thorvald II Agricultural Robotic System. Robotics, 6.10.3390/robotics6040024 ↗
78. Terrain assessment for precision agriculture using vehicle dynamic modelling10.1016/j.biosystemseng.2017.06.025 ↗
79. Design of obstacle avoidance controller for agricultural tractor based on ROS
80. Xi, H., Li, W., Zhao, F., Chen, L., and Hu, Y. (2024, January 14–18). A Safe and Efficient Timed-Elastic-Band Planner for Unstructured Environments. Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, United Arab Emirates.10.1109/IROS58592.2024.10802811 ↗
81. On human-in-the-loop optimization of human–robot interaction10.1038/s41586-024-07697-2 ↗
82. Human-in-the-Loop Optimization of Wearable Robotic Devices to Improve Human–Robot Interaction: A Systematic Review10.1109/TCYB.2022.3224895 ↗
83. Liebers, C., Megarajan, P., Auda, J., Stratmann, T.C., Pfingsthorn, M., Gruenefeld, U., and Schneegass, S. (2024). Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot Training. Multimodal Technol. Interact., 8.10.3390/mti8030018 ↗