Beyond Binary Dialogues: Research and Development of a Linguistically Nuanced Conversation Design for Social Robots in Group–Robot Interactions
Source abstract
Abstract
In this paper, we detail the technical development of a conversation design that is sensitive to group dynamics and adaptable, taking into account the subtleties of linguistic variations between dyadic (i.e., one human and one agent) and group interactions in human–robot interaction (HRI) using the German language as a case study. The paper details the implementation of robust person and group detection with YOLOv5m and the expansion of knowledge databases using large language models (LLMs) to create adaptive multi-party interactions (MPIs) (i.e., group–robot interactions (GRIs)). We describe the use of LLMs to generate training data for socially interactive agents including social robots, as well as a self-developed synthesis tool, knowledge expander, to accurately map the diverse needs of different users in public spaces. We also outline the integration of a LLM as a fallback for open-ended questions not covered by our knowledge database, ensuring it can effectively respond to both individuals and groups within the MPI framework.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Social Robot Interaction and HRI · AI in Service Interactions · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Cologne Cobots Lab, TH Köln—University of Applied Sciences, 50679 Cologne, Germany
Funding
Federal Ministry of Education and Research of Germany
References (41)
Show source-provided references
1. Breazeal, C. (September, January 30). Social robots for health applications. Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA.10.1109/IEMBS.2011.6091328 ↗
2. Cerrato, L., and Campbell, N. (2017). Engagement in dialogue with social robots. Dialogues with Social Robots: Enablements, Analyses, and Evaluation, Springer.10.1007/978-981-10-2585-3_25 ↗
3. Jayaraman, S., Phillips, E.K., Church, D., and Riek, L.D. (2024, January 11–15). Social Robots in Healthcare: Characterizing Privacy Considerations. Proceedings of the Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Boulder, CO, USA. HRI ’24.10.1145/3610978.3640713 ↗
4. Webster, C., and Ivanov, S. (2020). Robots in Travel, Tourism and Hospitality: Key Findings from a Global Study, Zangador.
5. Hameed, I.A., Tan, Z., Thomsen, N.B., and Duan, X. (2016, January 24–28). User Acceptance of Social Robots: A Case Study. Proceedings of the International Conference on Advances in Computer-Human Interaction, Venice, Italy.
6. Williams, M.A. (2012, January 29–31). Robot Social Intelligence. Proceedings of the 4th International Conference on Social Robotics (ICSR 2012), Chengdu, China. Lecture Notes in Computer Science.10.1007/978-3-642-34103-8_5 ↗
7. Correia, F., Melo, F.S., and Paiva, A. (2019, January 11–14). Group Intelligence on Social Robots. Proceedings of the 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Republic of Korea.10.1109/HRI.2019.8673089 ↗
8. Sabanovic, S., Michalowski, M.P., and Simmons, R. (2006, January 27–29). Robots in the wild: Observing human-robot social interaction outside the lab. Proceedings of the 9th IEEE International Workshop on Advanced Motion Control, Istanbul, Turkey.
9. Designing robots in the wild: In situ prototype evaluation for a break management robot10.5898/JHRI.3.1.Sabanovic ↗
10. Oliveira, R., Arriaga, P., and Paiva, A. (2021). Human-robot interaction in groups: Methodological and research practices. Multimodal Technol. Interact., 5.10.3390/mti5100059 ↗
11. AI-based chatbots in customer service and their effects on user compliance10.1007/s12525-020-00414-7 ↗
12. Addlesee, A., Cherakara, N., Nelson, N., Hernández García, D., Gunson, N., Sieińska, W., Romeo, M., Dondrup, C., and Lemon, O. (2024, January 11–14). A Multi-party Conversational Social Robot Using LLMs. Proceedings of the Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, Boulder, CO, USA. HRI ’24.10.1145/3610978.3641112 ↗
13. Porcheron, M., Fischer, J.E., Reeves, S., and Sharples, S. (2018, January 21–26). Voice interfaces in everyday life. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3174214 ↗
14. Lin, C.W., Auvray, V., Elkind, D., Biswas, A., Fazel-Zarandi, M., Belgamwar, N., Chandra, S., Zhao, M., Metallinou, A., and Chung, T. (2020). Dialog Simulation with Realistic Variations for Training Goal-Oriented Conversational Systems. arXiv.
15. Bapat, R., Kucherbaev, P., and Bozzon, A. (2018, January 5–8). Effective crowdsourced generation of training data for chatbots natural language understanding. Proceedings of the Web Engineering: 18th International Conference, ICWE 2018, Proceedings 18, Cáceres, Spain.10.1007/978-3-319-91662-0_8 ↗
16. Parrish, A., Huang, W., Agha, O., Lee, S.H., Nangia, N., Warstadt, A., Aggarwal, K., Allaway, E., Linzen, T., and Bowman, S.R. (2021). Does Putting a Linguist in the Loop Improve NLU Data Collection?. arXiv.10.18653/v1/2021.findings-emnlp.421 ↗
17. Monta, M., and Androulakis, S. (2024, August 22). Intent-Utterance-Expander. Available online: https://github.com/miguelmota/intent-utterance-expander.
18. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2024). GPT-4 Technical Report. arXiv.
19. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The Llama 3 Herd of Models. arXiv.
20. Kwon, M., Hu, H., Myers, V., Karamcheti, S., Dragan, A., and Sadigh, D. (2023). Toward grounded social reasoning. arXiv.10.1109/ICRA57147.2024.10611218 ↗
21. Rosenbaum, A., Soltan, S., and Hamza, W. (2024, August 17). Using Large Language Models (Llms) to Synthesize Training Data. Available online: https://www.amazon.science/blog/using-large-language-models-llms-to-synthesize-training-data.
22. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv.
23. Paetzel-Prüsmann, M., and Kennedy, J. (2023, January 13–16). Improving a Robot’s Turn-Taking Behavior in Dynamic Multiparty Interactions. Proceedings of the Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm, Sweden. HRI ’23.10.1145/3568294.3580117 ↗
24. Müller, A., and Richert, A. (2023, January 28–31). No One is an Island-Investigating the Need for Social Robots (and Researchers) to Handle Multi-Party Interactions in Public Spaces. Proceedings of the 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Busan, Republic of Korea.10.1109/RO-MAN57019.2023.10309383 ↗
25. Fraune, M.R., Nishiwaki, Y., Sabanović, S., Smith, E.R., and Okada, M. (2017, January 6–9). Threatening Flocks and Mindful Snowflakes: How Group Entitativity Affects Perceptions of Robots. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria. HRI ’17.10.1145/2909824.3020248 ↗
26. Šabanović, S. (2020, January 10–13). We’re in This Together: Social Robots in Group, Organizational, and Community Interactions. Proceedings of the 8th International Conference on Human-Agent Interaction, Virtual Event.10.1145/3406499.3422314 ↗
27. I–C–E Framework: Concepts for Group Dynamics Research in Human-Robot Interaction: Revisiting Theory from Social Psychology on Ingroup Identification (I), Cohesion (C) and Entitativity (E)10.1007/s12369-020-00642-z ↗
28. Reig, S., Luria, M., Wang, J.Z., Oltman, D., Carter, E.J., and Steinfeld, A. (2020, January 23–26). Not Some Random Agent: Multi-person interaction with a personalizing service robot. Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, Cambridge, UK.10.1145/3319502.3374795 ↗
29. Faria, M., Melo, F.S., and Paiva, A. (2021, January 8–12). Understanding robots: Making robots more legible in multi-party interactions. Proceedings of the 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, BC, Canada.10.1109/RO-MAN50785.2021.9515485 ↗
30. Fraune, M.R., Šabanović, S., and Kanda, T. (2019). Human group presence, group characteristics, and group norms affect human-robot interaction in naturalistic settings. Front. Robot. AI, 6.10.3389/frobt.2019.00048 ↗
31. Robot-Centric Perception of Human Groups10.1145/3375798 ↗
32. Pathi, S.K., Kiselev, A., and Loutfi, A. (2022). Detecting Groups and Estimating F-Formations for Social Human–Robot Interactions. Multimodal Technol. Interact., 6.10.3390/mti6030018 ↗
33. Luber, M., Spinello, L., Silva, J., and Arras, K.O. (2012, January 7–12). Socially-aware robot navigation: A learning approach. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal.10.1109/IROS.2012.6385716 ↗
34. Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
36. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv.
37. Dinh, M.C., Tien, N.T., Tuyen, T.M., Xuan, N.V., Anh, P.T.Q., Bay, H.V., and Truong, X.T. (2023, January 27–29). Socially Aware Robot Navigation Framework: Automatic Detecting and Autonomously Approaching People in Unknown Dynamic Social Environments. Proceedings of the 2023 12th International Conference on Control, Automation and Information Sciences (ICCAIS), Hanoi, Vietnam.10.1109/ICCAIS59597.2023.10382306 ↗
38. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
39. Sutawika, L., Gao, L., Schoelkopf, H., Biderman, S., Tow, J., Abbasi, B., Fattori, B., Lovering, C., Phang, J., and Thite, A. (2024, August 18). EleutherAI/lm-Evaluation-Harness: Major Refactor. Available online: https://zenodo.org/records/10256836.
40. Jocher, G., Stoken, A., Borovec, J., Liu, C., Hogan, A., Diaconu, L., Ingham, F., Fang, J., Wang, M., and Gupta, N. (2024, August 07). Ultralytics/yolov5: V3.1—Bug Fixes and Performance Improvements. Available online: https://zenodo.org/records/4154370.
41. Sabharwal, N., and Agrawal, A. (2020). Introduction to Google Dialogflow. Cognitive Virtual Assistants Using Google Dialogflow: Develop Complex Cognitive Bots Using the Google Dialogflow Platform, Apress.10.1007/978-1-4842-5741-8 ↗