Can Gestural Filler Reduce User-Perceived Latency in Conversation with Digital Humans?
Source abstract
Abstract
The demand for a conversational system with digital humans has increased with the development of artificial intelligence. Latency can occur in such conversational systems because of natural language processing and network issues, which can deteriorate the user’s performance and the availability of the systems. There have been attempts to mitigate user-perceived latency by using conversational fillers in human–agent interaction and human–robot interaction. However, non-verbal cues, such as gestures, have received less attention in such attempts, despite their essential roles in communication. Therefore, we designed gestural fillers for the digital humans. This study examined the effects of whether the conversation type and gesture filler matched or not. We also compared the effects of the gestural fillers with conversational fillers. The results showed that the gestural fillers mitigate user-perceived latency and affect the willingness, impression, competence, and discomfort in conversations with digital humans.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Social Robot Interaction and HRI · AI in Service Interactions · Hearing Impairment and Communication
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Convergence Engineering, Pusan National University, Busan 46241, Korea
School of Computer Science and Engineering, Pusan National University, Busan 46241, Korea
Funding
Ministry of Culture, Sports and Tourism and Korea Creative Content Agency
References (53)
Show source-provided references
1. Yu, F., Jian, S., Shen, C., Xue, W., and Fu, Y. (2022, January 18–21). On the Issue of “Digital Human” in the context of digital transformation; On the Issue of “Digital Human” in the context of digital transformation. Proceedings of the 2022 International Conference on Culture-Oriented Science and Technology (CoST), Lanzhou, China.10.1109/CoST57098.2022.00060 ↗
2. Kim, D., and Jo, D. (2022). Effects on Co-Presence of a Virtual Human: A Comparison of Display and Interaction Types. Electronics, 11.10.3390/electronics11030367 ↗
3. Sylaiou, S., and Fidas, C. (2022). Virtual Humans in Museums and Cultural Heritage Sites. Appl. Sci., 12.10.3390/app12199913 ↗
4. Park, S., Kim, S.P., and Whang, M. (2021). Individual’s social perception of virtual avatars embodied with their habitual facial expressions and facial appearance. Sensors, 21.10.3390/s21175986 ↗
5. Jaiswal, S., Valstar, M., Kusumam, K., and Greenhalgh, C. (2019, January 2–5). Virtual human questionnaire for analysis of depression, anxiety and personality. Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, Paris, France.10.1145/3308532.3329469 ↗
6. Kotranza, A., Lok, B., Pugh, C.M., and Lind, D.S. (2009, January 14–18). Virtual Humans That Touch Back: Enhancing Nonverbal Communication with Virtual Humans through Bidirectional Touch. Proceedings of the 2009 IEEE Virtual Reality Conference, Lafayette, LA, USA.10.1109/VR.2009.4811019 ↗
7. Krogmeier, C., Mousas, C., and Whittinghill, D. (2019, January 23–27). Human, virtual human, bump! a preliminary study on haptic feedback. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.10.1109/VR.2019.8798139 ↗
8. Human-scale haptic interaction with a reactive virtual human in a real-time physics simulator10.1145/1146816.1146831 ↗
9. Beacco, A., Gallego, J., and Slater, M. (2020, January 25–28). Automatic 3d character reconstruction from frontal and lateral monocular 2d rgb views. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP40778.2020.9191091 ↗
10. Ning, Y., He, S., Wu, Z., Xing, C., and Zhang, L.J. (2019). A review of deep learning based speech synthesis. Appl. Sci., 9.10.3390/app9194050 ↗
11. Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., and Black, M.J. (2019, January 15–20). Capture, learning, and synthesis of 3D speaking styles. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01034 ↗
12. Speech gesture generation from the trimodal context of text, audio, and speaker identity10.1145/3414685.3417838 ↗
13. A Survey of Presence and Related Concepts10.1145/3134301 ↗
14. Jun, H., and Bailenson, J. (2020, January 9–13). Effects of behavioral and anthropomorphic realism on social influence with virtual humans in AR. Proceedings of the 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Recife, Brazil.10.1109/ISMAR-Adjunct51615.2020.00026 ↗
15. Nagendran, A., Pillat, R., Kavanaugh, A., Welch, G., and Hughes, C. (2013, January 6–8). AMITIES: Avatar-mediated interactive training and individualized experience system. Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology, Singapore.10.1145/2503713.2503731 ↗
16. Allouch, M., Azaria, A., and Azoulay, R. (2021). Conversational Agents: Goals, Technologies, Vision and Challenges. Sensors, 21.10.3390/s21248448 ↗
17. Embodied conversational agents: Representation and intelligence in user interfaces
18. Industry 5.0 and Society 5.0—Comparison, complementation and co-evolution10.1016/j.jmsy.2022.07.010 ↗
19. Mourtzis, D., Angelopoulos, J., and Panopoulos, N. (2022). A Literature Review of the Challenges and Opportunities of the Transition from Industry 4.0 to Society 5.0. Energies, 15.10.3390/en15176276 ↗
20. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): A randomized controlled trial10.2196/mental.7785 ↗
21. Hello Harlie: Enabling speech monitoring through chat-bot conversations
22. A mental health chatbot for regulating emotions (SERMO)-concept and usability test10.1109/TETC.2020.2974478 ↗
23. DeVault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., Georgila, K., Gratch, J., Hartholt, A., and Lhommet, M. (2014, January 5–9). SimSensei Kiosk: A virtual human interviewer for healthcare decision support. Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, Paris, France.
24. More Than Just Another Pretty Face: Embodied Conversational Agents10.1145/332051.332075 ↗
25. Ke, X., Hu, P., Yang, C., and Zhang, R. (2022). Human–Machine Multi-Turn Language Dialogue Interaction Based on Deep Learning. Micromachines, 13.10.3390/mi13030355 ↗
26. A Conversation History-Based Q&A Cache Mechanism for Multi-Layered Chatbot Services10.3390/app11219981 ↗
27. Bao, S., He, H., Wang, F., Wu, H., Wang, H., Wu, W., Wu, Z., Guo, Z., Lu, H., and Huang, X. (2021). Plato-xl: Exploring the large-scale pre-training of dialogue generation. arXiv.
28. Mathew, L., and Bindu, V. (2020, January 11–13). A review of natural language processing techniques for sentiment analysis using pre-trained models. Proceedings of the 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC), Erode, India.10.1109/ICCMC48092.2020.ICCMC-00064 ↗
29. Daher, K., Bardelli, Z., Casas, J., Mugellini, E., Khaled, O.A., and Lalanne, D. (2020, January 21–25). Embodied conversational agent for emotional recognition training. Proceedings of the ThinkMind, ACHI 2020: The Thirteenth International Conference on Advances in Computer-Human Interactions, Valencia, Spain.
30. The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents10.1080/088395199117360 ↗
31. Aneja, D., Hoegen, R., McDuff, D., and Czerwinski, M. (2021, January 8–13). Understanding conversational and expressive style in a multimodal embodied conversational agent. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445708 ↗
32. Fastspeech: Fast, robust and controllable text to speech
33. Language models are few-shot learners
34. Hoxmeier, J.A., and DiCesare, C. (2000, January 10–13). System response time and user satisfaction: An experimental study of browser-based applications. Proceedings of the Americas Conference on Information Systems, Long Beach, CA, USA.
35. Miller, R.B. (1968, January 9–11). Response time in man-computer conversational transactions. Proceedings of the December 9–11, 1968, Fall Joint Computer Conference, Part I, San Francisco, CA, USA.10.1145/1476589.1476628 ↗
36. How quickly should a communication robot respond? Delaying strategies and habituation effects10.1007/s12369-009-0012-8 ↗
37. A humanoid robot that pretends to listen to route guidance from a human10.1007/s10514-006-9007-6 ↗
38. Xu, D., Zhou, A., Zhang, X., Wang, G., Liu, X., An, C., Shi, Y., Liu, L., and Ma, H. (2020, January 10–14). Understanding operational 5G: A first measurement study on its coverage, performance and energy consumption. Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, New York, NY, USA.10.1145/3387514.3405882 ↗
39. Wigdor, N., de Greeff, J., Looije, R., and Neerincx, M.A. (2016, January 26–31). How to improve human–robot interaction with Conversational Fillers. Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.10.1109/ROMAN.2016.7745134 ↗
40. Ohshima, N., Kimijima, K., Yamato, J., and Mukawa, N. (September, January 31). A conversational robot with vocal and bodily fillers for recovering from awkward silence at turn-takings. Proceedings of the 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan.10.1109/ROMAN.2015.7333677 ↗
41. Boukaram, H.A., Ziadee, M., and Sakr, M.F. (2021, January 9–11). Mitigating the Effects of Delayed Virtual Agent Response Time Using Conversational Fillers. Proceedings of the 9th International Conference on Human-Agent Interaction, Virtual Event, Japan.10.1145/3472307.3484181 ↗
42. Argyle, M. (2013). Bodily Communication, Routledge.10.4324/9780203753835 ↗
43. The biological basis of rapport10.1207/s15327965pli0104_3 ↗
44. Nonverbal behavior in clinician—Patient interaction10.1016/S0962-1849(05)80049-6 ↗
45. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information10.1075/pc.7.1.03cas ↗
46. The mismatch between gesture and speech as an index of transitional knowledge10.1016/0010-0277(86)90053-3 ↗
47. Reporting mental health symptoms: Breaking down barriers to care with virtual human interviewers10.3389/frobt.2017.00051 ↗
48. Breaking bad behaviors: A new tool for learning classroom management using virtual reality10.3389/fict.2016.00026 ↗
49. The effect of time delay on emotion, arousal, and satisfaction in human–robot interaction10.1177/1541931215591094 ↗
50. Mukawa, N., Sasaki, H., and Kimura, A. (2014, January 25–29). How do verbal/bodily fillers ease embarrassing situations during silences in conversations?. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, UK.10.1109/ROMAN.2014.6926226 ↗
51. The contribution of kinesic illustrators toward the comprehension of verbal behavior within utterances10.1111/j.1468-2958.1978.tb00622.x ↗
52. Measurement of negative attitudes toward robots10.1075/is.7.3.14nom ↗
53. Carpinella, C.M., Wyman, A.B., Perez, M.A., and Stroessner, S.J. (2017, January 6–9). The robotic social attributes scale (RoSAS) development and validation. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria.10.1145/2909824.3020208 ↗