Vocal Communication Between Cobots and Humans to Enhance Productivity and Safety: Review and Discussion
Source abstract
Abstract
This paper explores strategies for fostering efficient vocal communication and collaboration between human workers and collaborative robots (cobots) in assembly processes. Vocal communication enables the division of attention of the worker, as it frees their visual attention and the worker’s hands, dedicated to the task at hand. Speech generation and speech recognition are pre-requisites for effective vocal communication. This study focuses on cobot assistive tasks, where the human is in charge of the work and performs the main tasks while the cobot assists the worker in various peripheral jobs, such as bringing tools, parts, or materials, and returning them or disposing of them, or screwing or packaging the products. A nuanced understanding is necessary for optimizing human–robot interactions and enhancing overall productivity and safety. Through a comprehensive review of the relevant literature and an illustrative example with worked scenarios, this manuscript identifies key factors influencing successful vocal communication and proposes practical strategies for implementation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Social Robot Interaction and HRI · Teleoperation and Haptic Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Industrial Engineering Department, Tel Aviv Afeka Academic College of Engineering, Tel Aviv 69988, Israel
Department of Technical Management of Industrial Systems, University of Padova, 35122 Padova, Italy
Industrial Engineering and Technology Management Department, Holon Institute of Technology, Holon 58102, Israel
References (78)
Show source-provided references
1. Intelligent Cobot Systems: Human-Cobot Collaboration in Manufacturing10.1007/s10845-023-02142-z ↗
2. Human Factors in Cobot Era: A Review of Modern Production Systems Features10.1007/s10845-022-01953-w ↗
3. Human Robot Collaboration for Enhancing Work Activities10.1177/00187208221077722 ↗
4. A Human-Oriented Design Process for Collaborative Robotics10.1080/0951192X.2022.2128222 ↗
5. A Communicative Perspective on Human–Robot Collaboration in Industry: Mapping Communicative Modes on Collaborative Scenarios
6. The Impact of the Physical and Social Embodiment of Voice User Interfaces on User Distraction10.1016/j.ijhcs.2022.102784 ↗
7. Do Head-Mounted Augmented Reality Devices Affect Muscle Activity and Eye Strain of Utility Workers Who Do Procedural Work? Studies of Operators and Manhole Workers10.1177/0018720820943710 ↗
8. Heydaryan, S., Suaza Bedolla, J., and Belingardi, G. (2018). Safety Design and Development of a Human-Robot Collaboration Assembly Process in the Automotive Industry. Appl. Sci., 8.10.3390/app8030344 ↗
9. Review of Task Allocation for Human-Robot Collaboration in Assembly10.1080/0951192X.2023.2204467 ↗
10. An Empirical Study on Workers’ Preferences in Human–Robot Task Assignment in Industrial Assembly Systems10.1109/THMS.2022.3230667 ↗
11. Empirical Study for Human Engagement in Collaborative Robot Programming
12. Scalise, R., Rosenthal, S., and Srinivasa, S. (2017, January 6–9). Natural Language Explanations in Human-Collaborative Systems. Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria.10.1145/3029798.3034809 ↗
13. Kontogiorgos, D. (2023). Utilising Explanations to Mitigate Robot Conversational Failures. arXiv.
14. Zieliński, K., Walas, K., Heredia, J., and Kjærgaard, M.B. (2021, January 8–12). A Study of Cobot Practitioners Needs for Augmented Reality Interfaces in the Context of Current Technologies. Proceedings of the 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, Canada.10.1109/RO-MAN50785.2021.9515406 ↗
15. Pascher, M., Kronhardt, K., Franzen, T., Gruenefeld, U., Schneegass, S., and Gerken, J. (2022). My Caregiver the Cobot: Comparing Visualization Techniques to Effectively Communicate Cobot Perception to People with Physical Impairments. Sensors, 22.10.3390/s22030755 ↗
16. Eimontaite, I., Cameron, D., Rolph, J., Mokaram, S., Aitken, J.M., Gwilt, I., and Law, J. (2022). Dynamic Graphical Instructions Result in Improved Attitudes and Decreased Task Completion Time in Human–Robot Co-Working: An Experimental Manufacturing Study. Sustainability, 14.10.3390/su14063289 ↗
17. Carriero, G., Calzone, N., Sileo, M., Pierri, F., Caccavale, F., and Mozzillo, R. (2023). Human-Robot Collaboration: An Augmented Reality Toolkit for Bi-Directional Interaction. Appl. Sci., 13.10.3390/app132011295 ↗
18. Zoomorphic Gestures for Communicating Cobot States10.1109/LRA.2021.3060416 ↗
19. Human-robot-interaction using cloud-based speech recognition systems10.1016/j.procir.2020.05.214 ↗
20. Orienting Auditory Attention through Vision: The Impact of Monaural Listening10.1163/22134808-bja10059 ↗
21. Resing, M. (2023). Industrial Cobot Sound Design Study: Audio Design Principles for Industrial Human-Robot Interaction. [Master’s Thesis, University of Twente].
22. Tran, N. (2020). Exploring Mixed Reality Robot Communication under Different Types of Mental Workload, Colorado School of Mines.10.1145/3371382.3377438 ↗
23. Telkes, P., Angleraud, A., and Pieters, R. (2024, January 8–11). Instructing Hierarchical Tasks to Robots by Verbal Commands. Proceedings of the 2024 IEEE/SICE International Symposium on System Integration (SII), Ha Long, Vietnam.10.1109/SII58957.2024.10417491 ↗
24. Purposeful Communication in Human–Robot Collaboration: A Review of Modern Approaches in Manufacturing10.1109/ACCESS.2022.3227049 ↗
25. Rusan, H.-A., and Mocanu, B. (2022, January 10–11). Human-Computer Interaction Through Voice Commands Recognition. Proceedings of the 2022 International Symposium on Electronics and Telecommunications, Timisoara, Romania.10.1109/ISETC56213.2022.10010253 ↗
26. Carr, C., Wang, P., and Wang, S. (2023). A Human-Friendly Verbal Communication Platform for Multi-Robot Systems: Design and Principles. UK Workshop on Computational Intelligence, Springer.10.1007/978-3-031-47508-5_45 ↗
27. McMillan, D., Jaber, R., Cowan, B.R., Fischer, J.E., Irfan, B., Cumbal, R., Zargham, N., and Lee, M. (2023, January 13–16). Human-Robot Conversational Interaction (HRCI). Proceedings of the Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm, Sweden.10.1145/3568294.3579954 ↗
28. Lee, K.M., Krishna, A., Zaidi, Z., Paleja, R., Chen, L., Hedlund-Botti, E., Schrum, M., and Gombolay, M. (2023, January 13–16). The Effect of Robot Skill Level and Communication in Rapid, Proximate Human-Robot Collaboration. Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm, Sweden.10.1145/3568162.3577002 ↗
29. Programming Cobots by Voice: A Human-Centered, Web-Based Approach10.1016/j.procir.2020.05.213 ↗
30. Sorgini, F., Farulla, G.A., Lukic, N., Danilov, I., Roveda, L., Milivojevic, M., Pulikottil, T.B., Carrozza, M.C., Prinetto, P., and Tolio, T. (2020, January 3–5). Tactile Sensing with Gesture-Controlled Collaborative Robot. Proceedings of the 2020 IEEE International Workshop on Metrology for Industry 4.0 & IoT, Rome, Italy.10.1109/MetroInd4.0IoT48571.2020.9138183 ↗
31. Motion Strategies for a Cobot in a Context of Intermittent Haptic Interface10.1115/1.4054509 ↗
32. Zurlo, D., Heitmann, T., Morlock, M., and De Luca, A. (June, January 29). Collision Detection and Contact Point Estimation Using Virtual Joint Torque Sensing Applied to a Cobot. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK.10.1109/ICRA48891.2023.10160661 ↗
33. Inducing Self-Motion Sensations with Haptic Feedback: State-of-the-Art and Perspectives on “Haptic Motion”10.1109/TOH.2023.3279267 ↗
34. Collaborative Robots in Manufacturing and Assembly Systems: Literature Review and Future Research Agenda10.1007/s10845-023-02137-w ↗
35. Achieving productivity and operator well-being: A dynamic task allocation strategy for collaborative assembly systems in Industry 5.0
36. Schreiter, T., Morillo-Mendez, L., Chadalavada, R.T., Rudenko, A., Billing, E., Magnusson, M., Arras, K.O., and Lilienthal, A.J. (2023, January 28–31). Advantages of Multimodal versus Verbal-Only Robot-to-Human Communication with an Anthropomorphic Robotic Mock Driver. Proceedings of the 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Busan, South Korea.10.1109/RO-MAN57019.2023.10309629 ↗
37. Rautiainen, S., Pantano, M., Traganos, K., Ahmadi, S., Saenz, J., Mohammed, W.M., and Martinez Lastra, J.L. (2022). Multimodal Interface for Human–Robot Collaboration. Machines, 10.10.3390/machines10100957 ↗
38. Nonverbal Cues in Human–Robot Interaction: A Communication Studies Perspective10.1145/3570169 ↗
39. Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality10.1109/ACCESS.2021.3071364 ↗
40. Attention Bottlenecks for Multimodal Fusion
41. Significant Applications of Cobots in the Field of Manufacturing10.1016/j.cogr.2022.10.001 ↗
42. Cobotics: The Evolving Roles and Prospects of Next-Generation Collaborative Robots in Industry 5.010.1155/2024/2918089 ↗
43. Integrating collaboration scenarios and workforce individualization in collaborative assembly line balancing10.1016/j.ijpe.2024.109450 ↗
44. Overview of Ergonomics and Safety Aspects of Human-Cobot Interaction in the Manufacturing Industry
45. Adamini, R., Antonini, N., Borboni, A., Medici, S., Nuzzi, C., Pagani, R., and Tonola, C. (2021, January 18–22). User-friendly human-robot interaction based on voice commands and visual systems. Proceedings of the 2021 24th International Conference on Mechatronics Technology (ICMT), Singapore.10.1109/ICMT53429.2021.9687192 ↗
46. Programming cobots by voice: A pragmatic, web-based approach10.1080/0951192X.2022.2148754 ↗
47. Rouillard, J., and Vannobel, J.M. (2023). Multimodal Interaction for Cobot Using MQTT. Multimodal Technol. Interact., 7.10.3390/mti7080078 ↗
48. Su, H., Qi, W., Chen, J., Yang, C., Sandoval, J., and Laribi, M.A. (2023). Recent advancements in multimodal human–robot interaction. Front. Neurorobotics, 17.10.3389/fnbot.2023.1084000 ↗
49. Ferrari, D., Pupa, A., Signoretti, A., and Secchi, C. (2023). Safe Multimodal Communication in Human-Robot Collaboration. In International Workshop on Human-Friendly Robotics, Springer Nature.10.1007/978-3-031-55000-3_11 ↗
50. D’Attanasio, S., Alabert, T., Francis, C., and Studzinska, A. (2024, January 27–29). Exploring Multimodal Interactions with a Robot Assistant in an Assembly Task: A Human-Centered Design Approach. Proceedings of the 8th International Conference on Human Computer Interaction Theory and Applications, Rome, Italy.10.5220/0012570800003660 ↗
51. Abdulrazzaq, A.Z., Ali, Z.G., Al-Ani AR, M., Khaleel, B.M., Alsalame, S., Snovyda, V., and Kanbar, A.B. (2024, January 24–26). Evaluation of Voice Interface Integration with Arduino Robots in 5G Network Frameworks. Proceedings of the 2024 36th Conference of Open Innovations Association (FRUCT), Lappeenranta, Finland.10.23919/FRUCT64283.2024.10749856 ↗
52. Talk to your Cobot: Faster and more efficient error-handling in a robotic system with a multi-modal Conversational Agent
53. Asha, C.S., and D’Souza, J.M. (2024, January 16–17). Voice-Controlled Object Pick and Place for Collaborative Robots Employing the ROS2 Framework. Proceedings of the 2024 International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE), Shivamogga, India.
54. Younes, R., Elisei, F., Pellier, D., and Bailly, G. (2024, January 22–24). Impact of verbal instructions and deictic gestures of a cobot on the performance of human coworkers. Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Nancy, France.10.1109/Humanoids58906.2024.10769925 ↗
55. Siwach, G., and Li, C. (2023, January 12–14). Enhancing Human Cobot Interaction using Natural Language Processing. Proceedings of the 2023 IEEE 4th International Multidisciplinary Conference on Engineering Technology (IMCET), Beirut, Lebanon.10.1109/IMCET59736.2023.10368263 ↗
56. Siwach, G., and Li, C. (2024, January 6–8). Unveiling the Potential of Natural Language Processing in Collaborative Robots (Cobots): A Comprehensive Survey. Proceedings of the 2024 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.10.1109/ICCE59016.2024.10444393 ↗
57. Gkournelos, C., Konstantinou, C., Angelakis, P., Michalos, G., and Makris, S. (2023). Enabling Seamless Human-Robot Collaboration in Manufacturing Using LLMs. European Symposium on Artificial Intelligence in Manufacturing, Springer Nature.10.1007/978-3-031-57496-2_9 ↗
58. Lakhnati, Y., Pascher, M., and Gerken, J. (2024). Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation. Front. Robot. AI, 11.10.3389/frobt.2024.1347538 ↗
59. Kim, C.Y., Lee, C.P., and Mutlu, B. (2024, January 11–15). Understanding large-language model (llm)-powered human-robot interaction. Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Boulder, CO, USA.10.1145/3610977.3634966 ↗
60. Improved trust in human-robot collaboration with ChatGPT10.1109/ACCESS.2023.3282111 ↗
61. An LLM-based vision and language cobot navigation approach for Human-centric Smart Manufacturing10.1016/j.jmsy.2024.04.020 ↗
62. A survey on integration of large language models with intelligent robots10.1007/s11370-024-00550-5 ↗
63. Zhang, C., Chen, J., Li, J., Peng, Y., and Mao, Z. (2023). Large language models for human-robot interaction: A review. Biomim. Intell. Robot., 3.10.1016/j.birob.2023.100131 ↗
64. Jeong, H., Lee, H., Kim, C., and Shin, S. (2024). A Survey of Robot Intelligence with Large Language Models. Appl. Sci., 14.10.3390/app14198868 ↗
65. How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey
66. Real-world robot applications of foundation models: A review10.1080/01691864.2024.2408593 ↗
67. Research on the acceptance of collaborative robots for the industry 5.0 era—The mediating effect of perceived competence and the moderating effect of robot use self-efficacy10.1016/j.ergon.2023.103455 ↗
68. Nenna, F., Zanardi, D., Maria Orlando, E., Mingardi, M., Buodo, G., and Gamberini, L. (2024, January 26–28). Addressing Trust and Negative Attitudes Toward Robots in Human-Robot Collaborative Scenarios: Insights from the Industrial Work Setting. Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments, Crete, Greece.10.1145/3652037.3663905 ↗
69. Pietrantoni, L., Favilla, M., Fraboni, F., Mazzoni, E., Morandini, S., Benvenuti, M., and De Angelis, M. (2024). Integrating collaborative robots in manufacturing, logistics, and agriculture: Expert perspectives on technical, safety, and human factors. Front. Robot. AI, 11.10.3389/frobt.2024.1342130 ↗
70. Cobots in Industry 4.0: A Roadmap for Future Practice Studies on Human–Robot Collaboration10.1109/THMS.2021.3092684 ↗
71. Paliga, M. (2023). The relationships of human-cobot interaction fluency with job performance and job satisfaction among Cobot operators—The moderating role of workload. Int. J. Environ. Res. Public Health, 20.10.3390/ijerph20065111 ↗
72. Fraboni, F., Brendel, H., and Pietrantoni, L. (2023). Evaluating organizational guidelines for enhancing psychological well-being, safety, and performance in technology integration. Sustainability, 15.10.3390/su15108113 ↗
73. Application, development and future opportunities of collaborative robots (cobots) in manufacturing: A literature review10.1080/10447318.2022.2041907 ↗
74. Optimizing pick-place operations: Leveraging k-means for visual object localization and decision-making in collaborative robots
75. Deploying cobots in collaborative systems: Major considerations and productivity analysis10.1080/00207543.2020.1870758 ↗
76. Human-Robot Collaboration: An analysis of worker’s performance10.1016/j.procs.2022.01.355 ↗
77. Safety assurance mechanisms of collaborative robotic systems in manufacturing10.1016/j.rcim.2020.102022 ↗
78. Enhancing Human-Robot Collaboration in Industry 4.0 with AI-driven HRI10.52783/pst.196 ↗