The Development of an Emotional Embodied Conversational Agent and the Evaluation of the Effect of Response Delay on User Impression
Source abstract
Abstract
Embodied conversational agents (ECAs) are autonomous interaction interfaces designed to communicate with humans. This study investigates the impact of response delays and emotional facial expressions of ECAs on user perception and engagement. The motivation for this study stems from the growing integration of ECAs in various sectors, where their ability to mimic human-like interactions significantly enhances user experience. To this end, we developed an ECA with multimodal emotion recognition, both with voice and facial feature recognition and emotional facial expressions of the agent avatar. The system generates answers in real time based on media content. The development was supported by a case study of artwork images with the agent playing the role of a museum curator, where the user asks the agent for information on the artwork. We evaluated the developed system in two aspects. First, we investigated how the delay in an agent’s responses influences user satisfaction and perception. Secondly, we explored the role of emotion in an ECA’s face in shaping the user’s perception of responsiveness. The results showed that the longer response delay negatively impacted the user’s perception of responsiveness when the ECA did not express emotion, while the emotional expression improved the responsiveness perception.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Social Robot Interaction and HRI · Color perception and design
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Engineering, Tohoku University, Sendai 980-8579, Japan
École Centrale de Lyon, Av. Guy de Collongue, 69130 Écully, France
Funding
JSPS KAKENHI
References (54)
Show source-provided references
1. Embodied conversational interface agents10.1145/332051.332075 ↗
2. Embodied conversational agents: Representation and intelligence in user interfaces
3. Allouch, M., Azaria, A., and Azoulay, R. (2021). Conversational agents: Goals, technologies, vision and challenges. Sensors, 21.10.3390/s21248448 ↗
4. Mobile embodied conversational agent for task specific applications
5. Miyake, S., and Ito, A. (2012, January 3–6). A spoken dialogue system using virtual conversational agent with augmented reality. Proceedings of the 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA-ASC), Hollywood, CA, USA.
6. Fontecha, J., González, I., and Salas-Seguín, A. (2019). Using Conversational Assistants and Connected Devices to Promote a Responsible Energy Consumption at Home. Proceedings, 31.10.3390/proceedings2019031032 ↗
7. André, E. (2008). Design and evaluation of embodied conversational agents for educational and advisory software. Handbook of Conversation Design for Instructional Applications, IGI Global.10.4018/978-1-59904-597-9.ch020 ↗
8. Craig, P., Roa-Seïler, N., Rosano, F., and Díaz, M. (August, January 29). The role of embodied conversational agents in collaborative face to face computer supported learning games. Proceedings of the 26th International Conference on System Research, Informatics & Cybernetics, Baden-Baden, Germany.
9. Changing stigmatizing attitudes to mental health via education and contact with embodied conversational agents10.1016/j.chb.2017.03.071 ↗
10. Conversational agents in healthcare: A systematic review10.1093/jamia/ocy072 ↗
11. Bin Sawad, A., Narayan, B., Alnefaie, A., Maqbool, A., Mckie, I., Smith, J., Yuksel, B., Puthal, D., Prasad, M., and Kocaballi, A.B. (2022). A systematic review on healthcare artificial intelligent conversational agents for chronic conditions. Sensors, 22.10.3390/s22072625 ↗
12. Doumanis, I., and Smith, S. (2014, January 16). Evaluating the impact of embodied conversational agents (ECAs) attentional behaviors on user retention of cultural content in a simulated mobile environment. Proceedings of the 7th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze & Multimodality, Istanbul, Turkey.10.1145/2666642.2666650 ↗
13. Huget, M.P. (2003). Computational Model of Believable Conversational Agents. Communication in Multiagent Systems: Agent Communication Languages and Conversation Policies, Springer.10.1007/b12446 ↗
14. Wirzberger, M., Schmidt, R., Georgi, M., Hardt, W., Brunnett, G., and Rey, G.D. (2019). Effects of system response delays on elderly humans’ cognitive performance in a virtual training scenario. Sci. Rep., 9.10.1038/s41598-019-44718-x ↗
15. Strömbergsson, S., Hjalmarsson, A., Edlund, J., and House, D. (2013, January 25–29). Timing responses to questions in dialogue. Proceedings of the Interspeech, Lyon, France.10.21437/Interspeech.2013-581 ↗
16. A simplest systematics for the organization of turn-taking for conversation10.1353/lan.1974.0010 ↗
17. Hara, K., Inoue, K., Takanashi, K., and Kawahara, T. (2019, January 15–19). Turn-Taking Prediction Based on Detection of Transition Relevance Place. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-1537 ↗
18. Yahagi, R., Chiba, Y., Nose, T., and Ito, A. (2022). Multimodal Dialogue Response Timing Estimation Using Dialogue Context Encoder. Conversational AI for Natural Human-Centric Interaction, Proceedings of the 12th International Workshop on Spoken Dialogue System Technology, IWSDS 2021, Singapore, 15–17 November 2021, Springer.10.1007/978-981-19-5538-9_9 ↗
19. Sakuma, J., Fujie, S., Zhao, H., and Kobayashi, T. (2023, January 20–24). Improving the response timing estimation for spoken dialogue systems by reducing the effect of speech recognition delay. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-1618 ↗
20. Switching pauses in cooperative and competitive conversations10.1016/0022-1031(84)90027-1 ↗
21. Devillers, L., Vasilescu, I., and Vidrascu, L. (2004, January 23–26). F0 and pause features analysis for anger and fear detection in real-life spoken dialogs. Proceedings of the Speech Prosody, Nara, Japan.10.21437/SpeechProsody.2004-47 ↗
22. Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., and Yan, H. (1999, January 15–20). Embodiment in conversational interfaces: Rea. Proceedings of the the SIGCHI Conference on Human Factors in Computing Systems, CHI ’99, New York, NY, USA.10.1145/302979.303150 ↗
23. Human-like communication in conversational agents: A literature review and research agenda10.1108/JOSM-06-2019-0175 ↗
24. On the design of and interaction with conversational agents: An organizing and assessing review of human-computer interaction research
25. The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents10.1080/088395199117360 ↗
26. Ochs, M., Pelachaud, C., and Sadek, D. (2008, January 12–16). An empathic virtual dialog agent to improve human-machine interaction. Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Estoril, Portugal.
27. Becker, C., Kopp, S., and Wachsmuth, I. (2004, January 14–16). Simulating the emotion dynamics of a multimodal conversational agent. Proceedings of the Tutorial and Research Workshop on Affective Dialogue Systems, Kloster Irsee, Germany.10.1007/978-3-540-24842-2_15 ↗
28. Generic personality and emotion simulation for conversational agents10.1002/cav.3 ↗
29. A formal model of emotions for an empathic rational dialog agent10.1007/s10458-010-9156-z ↗
30. Chiba, Y., Nose, T., Kase, T., Yamanaka, M., and Ito, A. (2018, January 12–14). An analysis of the effect of emotional speech synthesis on non-task-oriented dialogue system. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia.10.18653/v1/W18-5044 ↗
31. A study on a spoken dialogue system with cooperative emotional speech synthesis using acoustic and linguistic information
32. EmoSen: Generating sentiment and emotion controlled responses in a multimodal dialogue system10.1109/TAFFC.2020.3015491 ↗
33. The effect of multimodal emotional expression on responses to a digital human during a self-disclosure conversation: A computational analysis of user language10.1007/s10916-020-01624-4 ↗
34. Saha, T., Reddy, S., Das, A., Saha, S., and Bhattacharyya, P. (2022, January 10–15). A shoulder to cry on: Towards a motivational virtual assistant for assuaging mental agony. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.174 ↗
35. Jokinen, K., Homma, K., Matsumoto, Y., and Fukuda, K. (2021, January 13–15). Integration and interaction of trustworthy ai in a virtual coach—An overview of EU-Japan collaboration on eldercare. Proceedings of the Annual Conference of the Japanese Society for Artificial Intelligence, Yokohama, Japan.10.1007/978-3-030-96451-1_17 ↗
36. The avatar will see you now: Support from a virtual human provides socio-emotional benefits10.1016/j.chb.2022.107368 ↗
37. Funk, M., Cunningham, C., Kanver, D., Saikalis, C., and Pansare, R. (2020, January 21–22). Usable and acceptable response delays of conversational agents in automotive user interfaces. Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Virtual.10.1145/3409120.3410651 ↗
38. Gnewuch, U., Morana, S., Adam, M., and Maedche, A. (2018, January 23–28). Faster is not always better: Understanding the effect of dynamic response delays in human-chatbot interaction. Proceedings of the European Conference on Information Systems, Portsmouth, UK.
39. Heeman, P., and Lunsford, R. (2017, January 20–24). Turn-Taking Offsets and Dialogue Context. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-1495 ↗
40. Jolibois, S., Ito, A., and Nose, T. (2023, January 23–28). Multimodal Expressive Embodied Conversational Agent Design. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark.10.1007/978-3-031-35989-7_31 ↗
41. Baltrušaitis, T., Robinson, P., and Morency, L.P. (2016, January 7–10). Openface: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477553 ↗
42. Ekman, P., and Friesen, W.V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press.10.1037/t27734-000 ↗
43. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
44. Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. (2020, January 5–10). GoEmotions: A Dataset of Fine-Grained Emotions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual.10.18653/v1/2020.acl-main.372 ↗
45. Public speech-oriented guidance system with adult and child discrimination capability
46. Inoue, R., Kurosawa, Y., Mera, K., and Takezawa, T. (2011, January 26–28). A question-and-answer classification technique for constructing and managing spoken dialog system. Proceedings of the 2011 International Conference on Speech Database and Assessments (Oriental COCOSDA), Hsinchu, Taiwan.10.1109/ICSDA.2011.6085987 ↗
47. Navarretta, C. (2016, January 23–28). Mirroring facial expressions and emotions in dyadic conversations. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia.
48. Mimicry and expressiveness of an ECA in human-agent interaction: Familiarity breeds content!10.1186/s40469-016-0008-2 ↗
49. Switching pauses in adult-adult and child-child turn takings: An initial study10.1007/BF01068018 ↗
50. An analysis of switching pause duration as a paralinguistic feature in expressive dialogues10.1250/ast.30.376 ↗
51. Peras, D. (2018). Chatbot evaluation metrics. Economic and Social Development: Book of Proceedings, Varazdin Development and Entrepreneurship Agency.
52. Shared perceptual basis of emotional expressions and trustworthiness impressions from faces10.1037/a0014520 ↗
53. Marcus, A., and Wang, W. (2019). Evaluating Response Delay of Multimodal Interface in Smart Device. Design, User Experience, and Usability. Practice and Case Studies, Springer.
54. Asaka, S., Itoyama, K., and Nakadai, K. (2024, January 26–30). Improving Impressions of Response Delay in AI-based Spoken Dialogue Systems. Proceedings of the 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), Pasadena, CA, USA.10.1109/RO-MAN60168.2024.10731216 ↗