journal article·2021-03-11Open access

Vocal Synchrony of Robots Boosts Positive Affective Empathy

Shogo Nishimura, Takuya Nakamura, Wataru Sato, Masayuki Kanbara, Yuichiro Fujimoto, Hirokazu Kato, Norihiro Hagita

Applied Sciences· Volume 11· Issue 6

Abstract

Robots that can talk with humans play increasingly important roles in society. However, current conversation robots remain unskilled at eliciting empathic feelings in humans. To address this problem, we used a robot that speaks in a voice synchronized with human vocal prosody. We conducted an experiment in which human participants held positive conversations with the robot by reading scenarios under conditions with and without vocal synchronization. We assessed seven subjective responses related to affective empathy (e.g., emotional connection) and measured the physiological emotional responses using facial electromyography from the corrugator supercilii and zygomatic major muscles as well as the skin conductance level. The subjective ratings consistently revealed heightened empathic responses to the robot in the synchronization condition compared with that under the de-synchronizing condition. The physiological signals showed that more positive and stronger emotional arousal responses to the robot with synchronization. These findings suggest that robots that are able to vocally synchronize with humans can elicit empathic emotional responses.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Social Robot Interaction and HRI · Action Observation and Synchronization · Face Recognition and Perception

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Nara Institute of Science and Technology, Graduate School of Information Science, Ikoma, Nara 630-0192, Japan

Psychological Process Team, Robotics Project, BZP, RIKEN, Kyoto 619-0288, Japan

Art Science Department, Osaka University of Arts, Osaka 585-8555, Japan

Funding

JSPS Foundation · Research Complex Promotion Program

References (65)

Show source-provided references
1. Social robots for long-term interaction: A survey10.1007/s12369-013-0178-y ↗
2. Breazeal, C. (1998, January 26–30). A motivational system for regulating human-robot interaction. Proceedings of the 15th National Conference on Artificial Intellilgence, Madison, WI, USA.
3. Gockley, R., Bruce, A., Forlizzi, J., Michalowski, M., Mundell, A., Rosenthal, S., Sellner, B., Simmons, R., Snipes, K., and Schultz, A.C. (2005, January 2–6). Designing robots for long-term social interaction. Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, AB, Canada.10.1109/IROS.2005.1545303 ↗
4. The empathic brain: How, when and why?10.1016/j.tics.2006.08.008 ↗
5. Empathy present and future10.1080/00224545.2018.1477442 ↗
6. Human-Agent and Human-Robot Interaction Theory: Similarities to and Differences from Human-Human Interaction10.1007/978-3-642-25691-2_9 ↗
7. Cassell, J., Sullivan, J., Churchill, E., and Prevost, S. (2000). Embodied Conversational Agents, MIT Press.10.7551/mitpress/2697.001.0001 ↗
8. Towards human-like spoken dialogue systems10.1016/j.specom.2008.04.002 ↗
9. Nass, C.I., and Brave, S. (2005). Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship, MIT Press.
10. Interpersonal synchrony: A survey of evaluation methods across disciplines10.1109/T-AFFC.2012.12 ↗
11. The antecedents and consequences of human behavioral mimicry10.1146/annurev-psych-113011-143754 ↗
12. How interpersonal synchrony facilitates early prosocial behavior10.1016/j.copsyc.2017.08.009 ↗
13. The chameleon effect: The perception—Behavior link and social interaction10.1037/0022-3514.76.6.893 ↗
14. It’s all in the timing: Interpersonal synchrony increases affiliation10.1521/soco.2009.27.6.949 ↗
15. Mimicry and prosocial behavior10.1111/j.0963-7214.2004.01501012.x ↗
16. Emergence of structured interactions: From a theoretical model to pragmatic robotics10.1016/j.neunet.2009.01.005 ↗
17. Using the rhythm of nonverbal human—Robot interaction as a signal for learning10.1109/TAMD.2010.2097260 ↗
18. Delaherche, E., Boucenna, S., Karp, K., Michelet, S., Achard, C., and Chetouani, M. (2012). Social coordination assessment: Distinguishing between shape and timing. IAPR Workshop on Multimodal Pattern Recognition of Social Signals in Human-Computer Interaction, Springer.10.1007/978-3-642-37081-6_2 ↗
19. A robot learns the facial expressions recognition and face/non-face discrimination through an imitation game10.1007/s12369-014-0245-z ↗
20. Towards artificial empathy10.1007/s12369-014-0253-z ↗
21. Nadel, J., Simon, M., Canet, P., Soussignan, R., Blancard, P., Canamero, L., and Gaussier, P. (2006). Human responses to an expressive robot. Procs of the Sixth International Workshop on Epigenetic Robotics, Lund University.
22. Nishimura, S., Kimata, D., Sato, W., Kanbara, M., Fujimoto, Y., Kato, H., and Hagita, N. (2020). Positive Emotion Amplification by Representing Excitement Scene with TV Chat Agents. Sensors, 7.10.3390/s20247330 ↗
23. Prepin, K., and Pelachaud, C. (2011, January 28–30). Shared understanding and synchrony emergence synchrony as an indice of the exchange of meaning between dialog partners. Proceedings of the 3rd International Conference on Agents and Artificial Intelligence, Rome, Italy.
24. When my robot smiles at me: Enabling human-robot rapport via real-time head gesture mimicry10.1007/s12193-009-0028-2 ↗
25. The association of therapist empathy and synchrony in vocally encoded arousal10.1037/a0034943 ↗
26. Xiao, B., Georgiou, P.G., Imel, Z.E., Atkins, D.C., and Narayanan, S.S. (2013). Modeling therapist empathy and vocal entrainment in drug addiction counseling. INTERSPEECH, ISCA.10.21437/Interspeech.2013-251 ↗
27. More than reflections: Empathy in motivational interviewing includes language style synchrony between therapist and client10.1016/j.beth.2014.11.002 ↗
28. Effect of postural congruence on client’s perception of counselor empathy10.1037/0022-0167.30.2.158 ↗
29. Neurosciences, empathy, and healthy interpersonal relationships: Recent findings and implications for counseling psychology10.1037/cou0000021 ↗
30. Sadoughi, N., Pereira, A., Jain, R., Leite, I., and Lehman, J.F. (2017, January 6–9). Creating prosodic synchrony for a robot co-player in a speech-controlled game for children. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria.10.1145/2909824.3020244 ↗
31. Effects of echoic mimicry using hummed sounds on human–computer interaction10.1016/S0167-6393(02)00180-2 ↗
32. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications10.1037/h0043424 ↗
33. Current and potential methods for measuring emotion in tourism experiences: A review10.1080/13683500.2014.975679 ↗
34. What is an emotion?
35. Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology10.1016/S0006-3223(98)00275-3 ↗
36. Negotiated collusion: Modeling social language and its relationship effects in intelligent agents10.1023/A:1024026532471 ↗
37. Posture mirroring and rapport
38. Sun, X., and Nijholt, A. (2011). Multimodal embodied mimicry in interaction. Analysis of Verbal and Nonverbal Communication and Enactment. The Processing Issues, Springer.10.1007/978-3-642-25775-9_14 ↗
39. Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome10.1037/a0023419 ↗
40. Evaluating the engagement with social robots10.1007/s12369-015-0298-7 ↗
41. Chaspari, T., and Lehman, J.F. (2016, January 8–12). An Acoustic Analysis of Child-Child and Child-Robot Interactions for Understanding Engagement during Speech-Controlled Computer Games. Proceedings of the Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA.10.21437/Interspeech.2016-85 ↗
42. Speech convergence and speech evaluation in fact-finding interviews10.1111/j.1468-2958.1984.tb00043.x ↗
43. Delaherche, E., and Chetouani, M. (2010, January 29). Multimodal coordination: Exploring relevant features and measures. Proceedings of the 2nd International Workshop on Social Signal Processing, Florence, Italy.10.1145/1878116.1878131 ↗
44. Accent mobility: A model and some data
45. The language of intergroup distinctiveness
46. Bilous, F.R., and Krauss, R.M. (1988). Dominance and accommodation in the conversational behaviours of same-and mixed-gender dyads. Lang. Commun.10.1016/0271-5309(88)90016-X ↗
47. Talk and silence sequences in informal conversations III: Interspeaker influence10.1111/j.1468-2958.1981.tb00564.x ↗
48. Lee, A., and Kawahara, T. (2009, January 4–7). Recent development of open-source speech recognition engine julius. Proceedings of the APSIPA ASC 2009: Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference. Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference, Sapporo, Japan.
49. SPTK Working Group (2015). Examples for Using Speech Signal Processing Toolkit Ver. 3.9, SourceForge.
50. HOYA Corporation (2021, March 11). Speech Sythesis voiceText Web API. Available online: http://voicetext.jp/products/vt-webapi/.
51. “Mora method” for objective evaluation of severity of spasmodic dysphonia10.5112/jjlp.38.176 ↗
52. ATR Japanese speech database as a tool of speech recognition and synthesis10.1016/0167-6393(90)90011-W ↗
53. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences10.3758/BF03193146 ↗
54. Gouaillier, D., Hugel, V., Blazevic, P., Kilner, C., Monceaux, J., Lafourcade, P., Marnier, B., Serre, J., and Maisonnier, B. (2008). The nao humanoid: A combination of performance and affordability. arXiv.
55. Affect grid: A single-item scale of pleasure and arousal10.1037/0022-3514.57.3.493 ↗
56. Characteristics of multimedia counseling: A study of an interactive TV system10.4992/jjpsy.68.9 ↗
57. Silent disco: Dancing in synchrony leads to elevated pain thresholds and social closeness10.1016/j.evolhumbehav.2016.02.004 ↗
58. Smith, J. (2000). GrandChair: Conversational Collection of Family Stories. [Master’s Thesis, MIT Media Arts & Sciences].
59. Facial muscle activation patterns in healthy male humans: A multi-channel surface EMG study10.1016/j.jneumeth.2009.12.019 ↗
60. Enhanced facial EMG activity in response to dynamic facial expressions10.1016/j.ijpsycho.2008.06.001 ↗
61. Soderberg, G.L. (1992). Selected Topics in Surface Electromyography for Use in the Occupational Setting: Expert Perspectives, US Department of Health and Human Services, Public Health Service, Centers for Disease Control, National Institute for Occupational Safety and Health.
62. Empirical comparison of univariate and multivariate analysis of variance procedures10.1037/h0031323 ↗
63. Your robot therapist will see you now: Ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy10.2196/13216 ↗
64. Vocal synchrony in psychotherapy10.1521/jscp.2014.33.5.481 ↗
65. Synchrony in psychotherapy: A review and an integrative framework for the therapeutic alliance10.3389/fpsyg.2016.00862 ↗