Comparing the Effectiveness of Speech and Physiological Features in Explaining Emotional Responses during Voice User Interface Interactions
Source abstract
Abstract
The rapid rise of voice user interface technology has changed the way users traditionally interact with interfaces, as tasks requiring gestural or visual attention are swapped by vocal commands. This shift has equally affected designers, required to disregard common digital interface guidelines in order to adapt to non-visual user interaction (No-UI) methods. The guidelines regarding voice user interface evaluation are far from the maturity of those surrounding digital interface evaluation, resulting in a lack of consensus and clarity. Thus, we sought to contribute to the emerging literature regarding voice user interface evaluation and, consequently, assist user experience professionals in their quest to create optimal vocal experiences. To do so, we compared the effectiveness of physiological features (e.g., phasic electrodermal activity amplitude) and speech features (e.g., spectral slope amplitude) to predict the intensity of users’ emotional responses during voice user interface interactions. We performed a within-subjects experiment in which the speech, facial expression, and electrodermal activity responses of 16 participants were recorded during voice user interface interactions that were purposely designed to elicit frustration and shock, resulting in 188 analyzed interactions. Our results suggest that the physiological measure of facial expression and its extracted feature, automatic facial expression-based valence, is most informative of emotional events lived through voice user interface interactions. By comparing the unique effectiveness of each feature, theoretical and practical contributions may be noted, as the results contribute to voice user interface literature while providing key insights favoring efficient voice user interface evaluation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Multisensory perception and integration · Neuroscience and Music Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Tech3Lab, HEC Montréal, Montréal, QC H3T 1T7, Canada
Faculty of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
Deloitte, Montréal, QC H3T 1T7, Canada
Funding
NSERC Industrial Research Chair & Deloitte Digital
References (100+)
Show source-provided references
1. Murad, C., and Munteanu, C. (2020, January 25–30). Designing Voice Interfaces: Back to the (Curriculum) Basics. Proceedings of the CHI ‘20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376522 ↗
2. A validation of ergonomic criteria for the evaluation of human-computer interfaces10.1080/10447319209526035 ↗
3. Nielsen, J. (1994, January 24–28). Usability inspection methods. Proceedings of the CHI94: ACM Conference on Human Factors in Computer Systems, Boston, MA, USA.10.1145/259963.260531 ↗
4. Statista (2021, July 10). The Most Important Voice Platforms in 2020. Available online: https://www.statista.com/chart/22314/voice-platform-ranking/.
5. Improving the Usability of Voice User Interfaces: A New Set of Ergonomic Criteria10.1007/978-3-030-49760-6_8 ↗
6. Seaborn, K., and Urakami, J. (2021, January 8). Measuring Voice UX Quantitatively: A Rapid Review. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohoma, Japan.10.1145/3411763.3451712 ↗
7. Alves, R., Valente, P., and Nunes, N.J. (2014, January 26–30). The state of user experience evaluation practice. Proceedings of the NordiCHI 2014: The 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, Helsinki, Finland.10.1145/2639189.2641208 ↗
8. Usability testing of spoken conversational systems
9. Explicit and implicit antecedents of users’ behavioral beliefs in information systems: A neuropsychological investigation10.2753/MIS0742-1222300407 ↗
10. Den Uyl, M.J., and Van Kuilenburg, H. (September, January 30). The Facereader: Online Facial Expression Recognition. Proceedings of the Measuring Behavior 2005, Wageningen, The Netherlands.
11. A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments
12. The state of speech in HCI: Trends, themes and challenges10.1093/iwc/iwz016 ↗
13. Lopatovska, I., and Williams, H. (2018, January 11–15). Personification of the Amazon Alexa: BFF or a mindless companion. Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, New Brunswick, NJ, USA.10.1145/3176349.3176868 ↗
14. Garg, R., and Moreno, C. (2019, January 20). Exploring Everyday Sharing Practices of Smart Speakers. Proceedings of the IUI Workshops, Los Angeles, CA, USA.
15. Sciuto, A., Saini, A., Forlizzi, J., and Hong, J.I. (2018, January 9–13). “Hey Alexa, What’s Up?”. Proceedings of the Designing Interactive Systems Conference 2018—DIS ’18, Hong Kong, China.10.1145/3196709.3196772 ↗
16. User interactions with “Alexa” in public academic space10.1002/pra2.2018.14505501034 ↗
17. An investigation of information systems use patterns: Technological events as triggers, the effect of time, and consequences for performance10.25300/MISQ/2013/37.4.08 ↗
18. Psychophysiological Measures of Human Cognitive States Applied in Human Computer Interaction10.1016/j.procs.2011.01.016 ↗
19. Beyond Cognition and Affect: Sensing the Unconscious10.1080/0144929X.2014.912353 ↗
20. The voice conveys emotion in ten globalized cultures and one remote village in Bhutan10.1037/emo0000100 ↗
21. Communication of emotions in vocal expression and music performance: Different channels, same code?10.1037/0033-2909.129.5.770 ↗
22. Voice-only communication enhances empathic accuracy10.1037/amp0000147 ↗
23. The expression and recognition of emotions in the voice across five nations: A lens model analysis based on acoustic features10.1037/pspi0000066 ↗
24. Laughing, smiling, and talking: Relation to sleeping and social context in humans10.1111/j.1439-0310.1989.tb00536.x ↗
25. Tao, J., Tan, T., and Picard, R.W. (2005). Real-life emotion representation and detection in call centers data. Affective Computing and Intelligent Interaction, Springer.10.1007/11573548 ↗
26. Lausen, A., and Hammerschmidt, K. (2020). Emotion recognition and confidence ratings predicted by vocal stimulus type and prosodic parameters. Humanit. Soc. Sci. Commun., 7.10.1057/s41599-020-0499-z ↗
27. Vocal communication of emotion
28. Tahon, M., Degottex, G., and Devillers, L. (2012, January 25). Usual voice quality features and glottal features for emotional valence detection. Proceedings of the 6th International Conference on Speech Prosody, Shanghai, China.10.21437/SpeechProsody.2012-173 ↗
29. Shilker, T.S. (2009). Analysis of Affective Expression in Speech. [Ph.D. Thesis, Cambridge University].
30. Vocal Expression and Perception of Emotion10.1111/1467-8721.00013 ↗
31. Li, S.Z., and Jain, A. (2009). Fundamental Frequency, Pitch, F0. Encyclopedia of Biometrics, Springer.10.1007/978-3-642-27733-7 ↗
32. Suitability of dysphonia measurements for telemonitoring of Parkinson’s disease10.1109/TBME.2008.2005954 ↗
33. Developing a large scale population screening tool for the assessment of Parkinson’s disease using telephone-quality voice10.1121/1.5100272 ↗
34. Mannepalli, K., Sastry, P.N., and Suman, M. Emotion recognition in speech signals using optimization based multi-SVNN classifier. J. King Saud Univ. Comput. Inf. Sci. 2018, in press.
35. Toh, A.M., Togneri, R., and Nordholm, S. (2005). Spectral entropy as speech features for speech recognition. Proc. PEECS, 1.
36. Papakostas, M., Siantikos, G., Giannakopoulos, T., Spyrou, E., and Sgouropoulos, D. (2017). Recognizing emotional states using speech information. GeNeDis 2016, Springer.10.1007/978-3-319-57348-9_13 ↗
37. A Comprehensive Review of Speech Emotion Recognition Systems10.1109/ACCESS.2021.3068045 ↗
38. Robinson, C., Obin, N., and Roebel, A. (2019, January 12). Sequence-to-sequence modelling of f0 for speech emotion conversion. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683865 ↗
39. Voice conversion for emotional speech: Rule-based synthesis with degree of emotion controllable in dimensional space10.1016/j.specom.2018.06.006 ↗
40. A circumplex model of affect10.1037/h0077714 ↗
41. Zhu, C., and Ahmad, W. (2019, January 5–8). Emotion recognition from speech to improve human-robot interaction. Proceedings of the IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), Fukuoka, Japan.10.1109/DASC/PiCom/CBDCom/CyberSciTech.2019.00076 ↗
42. B3-1 Analysis of User’s Speech Behavior Pattern after Correction: Focusing on Smartphone Voice User Interface
43. Zaman, B., and Shrimpton-Smith, T. (2006, January 14). The FaceReader: Measuring instant fun of use. Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, Oslo, Norway.10.1145/1182475.1182536 ↗
44. Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology10.1016/S0006-3223(98)00275-3 ↗
45. Towards a deeper understanding of system usage in organizations10.2307/25148815 ↗
46. Cacioppo, J.T., Tassinary, L.G., and Berntson, G.G. (2007). The electrodermal system. Handbook of Psychophysiology, Cambridge University Press.
47. Bethel, C.L., Salomon, K., Murphy, R.R., and Burke, J.L. (2007, January 26–29). Survey of psychophysiology measurements applied to human-robot interaction. Proceedings of the RO-MAN 2007—The 16th IEEE International Symposium on Robot and Human Interactive Communication, Jeju, Korea.10.1109/ROMAN.2007.4415182 ↗
48. Riedl, R., and Léger, P.M. (2016). Fundamentals of NeuroIS: Information Systems and the Brain, Springer. Studies in Neuroscience, Psychology and Behavioral Economics.10.1007/978-3-662-45091-8 ↗
49. Neurophysiological Correlates of Cognitive Absorption in an Enactive Training Context10.1016/j.chb.2014.02.011 ↗
50. Application strategies for neuroscience in information systems design science research
51. Giroux-Huppé, C., Sénécal, S., Fredette, M., Chen, S.L., Demolin, B., and Léger, P.-M. (2019). Identifying Psychophysiological Pain Points in the Online User Journey: The Case of Online Grocery, Springer.10.1007/978-3-030-23535-2_34 ↗
52. Lamontagne, C., Sénécal, S., Fredette, M., Chen, S.L., Pourchon, R., Gaumont, Y., De Grandpré, D., and Léger, P.M. (2019, January 26). User Test: How Many Users Are Needed to Find the Psychophysiological Pain Points in a Journey Map?. Proceedings of the International Conference on Human Interaction and Emerging Technologies, Nice, France.10.1007/978-3-030-25629-6_22 ↗
53. User Experience—A Research Agenda10.1080/01449290500330331 ↗
54. Boucsein, W. (2012). Electrodermal Activity, Springer.10.1007/978-1-4614-1126-0 ↗
55. Ekman, P., and Friesen, W.V. (1978). The Facial Action Coding System, Consulting Psychologists Press.10.1037/t27734-000 ↗
56. Leite, I., Henriques, R., Martinho, C., and Paiva, A. (2013, January 3–6). Sensors in the wild: Exploring electrodermal activity in child-robot interaction. Proceedings of the 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan.10.1109/HRI.2013.6483500 ↗
57. Castellano, G., Kessous, L., and Caridakis, G. (2008). Emotion recognition through multiple modalities: Face, body gesture, speech. Affect and Emotion in Human-Computer Interaction, Springer.10.1007/978-3-540-85099-1_8 ↗
58. Emotion regulation and mental health10.1111/j.1468-2850.1995.tb00036.x ↗
59. Greco, A., Marzi, C., Lanata, A., Scilingo, E.P., and Vanello, N. (2019, January 23–27). Combining Electrodermal Activity and Speech Analysis towards a more Accurate Emotion Recognition System. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, German.10.1109/EMBC.2019.8857745 ↗
60. Prasetio, B.H., Tamura, H., and Tanno, K. (2020, January 26–29). Embedded Discriminant Analysis based Speech Activity Detection for Unsupervised Stress Speech Clustering. Proceedings of the 2020 Joint 9th International Conference on Informatics, Electronics and Vision and 2020 4th International Conference on Imaging, Vision and Pattern Recognition, ICIEV and IcIVPR, Kitakyushu, Japan.10.1109/ICIEVicIVPR48672.2020.9306589 ↗
61. Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A., and Karpouzis, K. (2006, January 2–4). Modeling naturalistic affective states via facial and vocal expressions recognition. Proceedings of the ICMI’06: 8th International Conference on Multimodal Interfaces, Banff, AB, Canada.10.1145/1180995.1181029 ↗
62. Alshamsi, H., Kepuska, V., Alshamsi, H., and Meng, H. (2019, January 1–3). Automated Facial Expression and Speech Emotion Recognition App Development on Smart Phones using Cloud Computing. Proceedings of the 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference, IEMCON 2018, Vancouver, BC, Canada.10.1109/IEMCON.2018.8614831 ↗
63. Vocal affect expression: A review and a model for future research10.1037/0033-2909.99.2.143 ↗
64. The contribution of speech rate and pitch variation to the perception of vocal emotions in a German and an American sample10.1080/02699930126095 ↗
65. Davitz, J.R. (1964). The Communication of Emotional Meaning, Mcgraw Hill.
66. Speech pitch frequency as an emotional state indicator10.1109/TSMC.1975.5408480 ↗
67. Pereira, C. (2000, January 5–7). Dimensions of emotional meaning in speech. Proceedings of the ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, Newcastle, UK.
68. Cue utilization in emotion attribution from auditory stimuli10.1007/BF00992539 ↗
69. Schröder, M., Cowie, R., Douglas-Cowie, E., Westerdijk, M., and Gielen, S. (2001, January 3–7). Acoustic correlates of emotion dimensions in view of speech synthesis. Proceedings of the Seventh European Conference on Speech Communication and Technology, Aalborg, Denmark.10.21437/Eurospeech.2001-34 ↗
70. Effects of pitch and speech rate on personal attributions10.1037/0022-3514.37.5.715 ↗
71. Kehrein, R. (2002, January 11–13). The prosody of authentic emotions. Proceedings of the Speech Prosody 2002 International Conference, Aix-en-Provence, France.10.21437/SpeechProsody.2002-91 ↗
72. The long-term spectrum and perceived emotion10.1016/0167-6393(90)90055-E ↗
73. A dimensional approach to vocal expression of emotion10.1080/02699930441000445 ↗
74. Weitz, S. (1974). Acoustic concomitants of emotional dimensions: Judging affect from synthesized tone sequences. Nonverbal Communication, Oxford University Press.
75. Attitudinal meanings conveyed by intonation contours10.1177/002383096000300403 ↗
76. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., and Narayanan, S. (2004, January 13–14). Analysis of emotion recognition using facial expressions, speech and multimodal information. Proceedings of the 6th International Conference on Multimodal Interfaces, State College, PA, USA.10.1145/1027933.1027968 ↗
77. Busso, C., and Rahman, T. (2012, January 9–13). Unveiling the acoustic properties that describe the valence dimension. Proceedings of the Thirteenth Annual Conference of the International Speech Communication Association, Portland, OR, USA.10.21437/Interspeech.2012-124 ↗
78. Giroux, F., Léger, P.M., Brieugne, D., Courtemanche, F., Bouvier, F., Chen, S.L., Tazi, S., Rucco, E., Fredette, M., and Coursaris, C. (2021, January 24–29). Guidelines for Collecting Automatic Facial Expression Detection Data Synchronized with a Dynamic Stimulus in Remote Moderated User Tests. Proceedings of the International Conference on Human-Computer Interaction, Washngton, DC, USA.10.1007/978-3-030-78462-1_18 ↗
79. Vasseur, A., Léger, P.M., Courtemanche, F., Labonte-Lemoyne, E., Georges, V., Valiquette, A., Brieugne, D., Rucco, E., Coursaris, C., and Fredette, M. (2021, January 24–29). Distributed remote psychophysiological data collection for UX evaluation: A pilot project. Proceedings of the International Conference on Human-Computer Interaction, Virtual Event.10.1007/978-3-030-78462-1_19 ↗
80. Figner, B., and Murphy, R.O. (2011). Using skin conductance in judgment and decision making research. A Handbook of Process Tracing Methods for Decision Research, Psychology Press.
81. Courtemanche, F., Fredette, M., Senecal, S., Leger, P.M., Dufresne, A., Georges, V., and Labonte-Lemoyne, E. (2019). Method of and System for Processing Signals Sensed from a User. (No. 10,368,741), U.S. Patent.
82. Courtemanche, F., Léger, P.M., Fredette, M., and Sénécal, S. (2022). Cobalt—Bluebox: Système de Synchronisation et d’Acquisition Sans-Fil de Données Utilisateur Multimodales, HEC Montréal. Declaration of Invention No. AXE-0045.
83. Measuring emotion: The self-assessment manikin and the semantic differential10.1016/0005-7916(94)90063-9 ↗
84. Betella, A., and Verschure, P.F. (2016). The Affective Slider: A Digital Self-Assessment Scale for the Measurement of Human Emotions. PLoS ONE, 11.10.1371/journal.pone.0148037 ↗
85. Valence, arousal, and dominance ratings for facial stimuli10.1177/1747021819829012 ↗
86. The temporal dynamics of processing emotions from vocal, facial, and bodily expressions10.1016/j.neuroimage.2011.06.035 ↗
87. Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Busso, C., Deng, Z., Lee, S., and Narayanan, S. (2004, January 4–8). An acoustic study of emotions expressed in speech. Proceedings of the Eighth International Conference on Spoken Language Processing, Jeju, Korea.10.21437/Interspeech.2004-242 ↗
88. Skiendziel, T., Rösch, A.G., and Schultheiss, O.C. (2019). Assessing the convergent validity between the automated emotion recognition software Noldus FaceReader 7 and Facial Action Coding System Scoring. PLoS ONE, 14.10.1371/journal.pone.0223905 ↗
89. Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader10.1037/npe0000028 ↗
90. Cohn, J.F., and Kanade, T. (2007). Use of automated facial image analysis for measurement of emotion expression. Handbook of Emotion Elicitation and Assessment, Oxford University Press.10.1093/oso/9780195169157.003.0015 ↗
91. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial10.20982/tqmp.08.1.p023 ↗
92. The intraclass correlation coefficient as a measure of reliability10.2466/pr0.1966.19.1.3 ↗
93. A guideline of selecting and reporting intraclass correlation coefficients for reliability research10.1016/j.jcm.2016.02.012 ↗
94. Bland, J.M., and Altman, D.G. (1995). Multiple significance tests: The Bonferroni method. BMJ, 310.10.1136/bmj.310.6973.170 ↗
95. Cicchetti, D.V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess., 6.10.1037//1040-3590.6.4.284 ↗
96. The Body and the Brain: Measuring Skin Conductance Responses to Understand the Emotional Experience10.1177/1094428116681073 ↗
97. Facial expression arousal level modulates facial mimicry10.1016/j.ijpsycho.2010.02.008 ↗
98. Patel, S., Scherer, K.R., Sundberg, J., and Björkner, E. (2010, January 10–14). Acoustic markers of emotions based on voice physiology. Proceedings of the Conference: Speech Prosody, Chicago, IL, USA.10.21437/SpeechProsody.2010-239 ↗
99. Privacy Concerns for Use of Voice Activated Personal Assistant in the Public Space10.1080/10447318.2014.986642 ↗
100. Jiang, J., Hassan Awadallah, A., Jones, R., Ozertem, U., Zitouni, I., Gurunath Kulkarni, R., and Khan, O.Z. (2015, January 18–22). Automatic Online Evaluation of Intelligent Assistants. Proceedings of the 24th International Conference on World Wide Web—WWW’15, Florence, Italy.10.1145/2736277.2741669 ↗