Review of Automatic Estimation of Emotions in Speech
Source abstract
Abstract
Identification of emotions exhibited in utterances is useful for many applications, e.g., assisting with handling telephone calls or psychological diagnoses. This paper reviews methods to identify emotions from speech signals. We examine the information in speech that helps to estimate emotion, from points of view involving both production and perception. As machine approaches to recognize emotion in speech often have much in common with other speech tasks, such as automatic speaker verification and speech recognition, we compare such processes. Many methods of emotion recognition have been found in research on pattern recognition in other areas, e.g., image and text recognition, especially in recent methods for machine learning. We show that speech is very different compared to most other signals that can be recognized, and that emotion identification is different from other speech applications. This review is primarily aimed at non-experts (more algorithmic detail is present in the cited literature), but this presentation has much discussion for experts as well.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Speech and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Telecommunication Department, INRS, University of Quebec, 800 de la Gauchetiere West, Montreal, QC H5A 1K6, Canada
Funding
NSERC
References (100+)
Show source-provided references
1. Accuracy of deception judgments10.1207/s15327957pspr1003_2 ↗
2. Emotion recognition in human-computer interaction10.1109/79.911197 ↗
3. Emotional speech recognition: Resources, features, and methods10.1016/j.specom.2006.04.003 ↗
4. Survey on speech emotion recognition: Features, classification schemes, and databases10.1016/j.patcog.2010.09.020 ↗
5. Emotion recognition from speech: A review10.1007/s10772-011-9125-1 ↗
6. Speech emotion recognition using deep learning techniques: A review10.1109/ACCESS.2019.2936124 ↗
7. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends10.1145/3129340 ↗
8. Vocal communication of emotion: A review of research paradigms10.1016/S0167-6393(02)00084-5 ↗
9. A systematic review on affective computing: Emotion models, databases, and recent advances10.1016/j.inffus.2022.03.009 ↗
10. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers10.1016/j.specom.2019.12.001 ↗
11. Deep learning approaches for speech emotion recognition: State of the art and research challenges10.1007/s11042-020-09874-7 ↗
12. Mower, E., Metallinou, A., Lee, C.C., Kazemzadeh, A., Busso, C., Lee, S., and Narayanan, S. (2009, January 10–12). Interpreting ambiguous emotional expressions. Proceedings of the International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, The Netherlands.10.1109/ACII.2009.5349500 ↗
13. Speech emotion recognition using deep 1D & 2D CNN LSTM networks10.1016/j.bspc.2018.08.035 ↗
14. Challenges in real-life emotion annotation and machine learning based detection10.1016/j.neunet.2005.03.007 ↗
15. Schuller, B., Reiter, S., Muller, R., Al-Hames, M., Lang, M., and Rigoll, G. (2005, January 6–8). Speaker independent speech emotion recognition by ensemble classification. Proceedings of the IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands.
16. Antoniou, N., Katsamanis, A., Giannakopoulos, T., and Narayanan, S. (2023, January 4–10). Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.
17. A circumplex model of affect10.1037/h0077714 ↗
18. Four models of basic emotions: A review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt10.1177/1754073911410747 ↗
19. Plutchik, R. (1991). The Emotions, University Press of America.
20. Ortony, A., Clore, G.L., and Collins, A. (2022). The Cognitive Structure of Emotions, Cambridge University Press.10.1017/9781108934053 ↗
21. Toward a comprehensive theory of emotion10.1016/S0065-2601(08)60132-X ↗
22. Plutchik, R., and Kellerman, H. (2013). Theories of Emotion, Academic Press.
23. Acoustic profiles in vocal emotion expression10.1037/0022-3514.70.3.614 ↗
24. Emotions and speech: Some acoustical correlates10.1121/1.1913238 ↗
25. Jin, Q., Li, C., Chen, S., and Wu, H. (2015, January 19–24). Speech emotion recognition with acoustic and lexical features. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178872 ↗
26. Lee, C.M., Yildirim, S., Bulut, M., Kazemzadeh, A., Busso, C., Deng, Z., Lee, S., and Narayanan, S. (2004, January 4–8). Emotion recognition based on phoneme classes. Proceedings of the International Conference on Spoken Language Processing, Jeju Island, Republic of Korea.10.21437/Interspeech.2004-322 ↗
27. Speech emotion recognition considering nonverbal vocalization in affective conversations10.1109/TASLP.2021.3076364 ↗
28. CTNet: Conversational transformer network for emotion recognition10.1109/TASLP.2021.3049898 ↗
29. A survey of textual emotion recognition and its challenges10.1109/TAFFC.2021.3053275 ↗
30. Xu, H., Liu, B., Shu, L., and Yu, P.S. (2019). BERT post-training for review reading comprehension and aspect-based sentiment analysis. arXiv, 2324–2335.
31. Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion10.1121/1.405558 ↗
32. How to find trouble in communication10.1016/S0167-6393(02)00079-1 ↗
33. Speech emotion recognition using hidden Markov models10.1016/S0167-6393(03)00099-2 ↗
34. Burkhardt, F., and Sendlmeier, W.F. (2000, January 5–7). Verification of acoustical correlates of emotional speech using formant-synthesis. Proceedings of the ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, Northern Ireland, UK.
35. Hierarchical representation and estimation of prosody using continuous wavelet transform10.1016/j.csl.2016.11.001 ↗
36. Speech coding: A tutorial review10.1109/5.326413 ↗
37. Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., and Pantic, M. (2016, January 16). Avec 2016: Depression, mood, and emotion recognition workshop and challenge. Proceedings of the International Workshop on Audio/Visual Emotion Challenge, Amsterdam, The Netherlands.10.1145/2964284.2980532 ↗
38. Cultural differences in emotion: Differences in emotional arousal level between the East and the West10.1016/j.imr.2016.03.004 ↗
39. Compensation of nuisance factors for speaker and language recognition10.1109/TASL.2007.901823 ↗
40. Features and classifiers for emotion recognition from speech: A survey from 2000 to 201110.1007/s10462-012-9368-5 ↗
41. Toward detecting emotions in spoken dialogs10.1109/TSA.2004.838534 ↗
42. Lugger, M., and Yang, B. (2007, January 15–20). The relevance of voice quality features in speaker independent emotion recognition. Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, HI, USA.10.1109/ICASSP.2007.367152 ↗
43. Sun, R., Moore, E., and Torres, J.F. (2009, January 19–24). Investigating glottal parameters for differentiating emotional categories with similar prosodics. Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.10.1109/ICASSP.2009.4960632 ↗
44. Interdependencies among voice source parameters in emotional speech10.1109/T-AFFC.2011.14 ↗
45. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences10.1109/TASSP.1980.1163420 ↗
46. Spectral analysis of speech by linear prediction10.1109/TAU.1973.1162470 ↗
47. Yu, D., and Deng, L. (2016). Automatic Speech Recognition, Springer.10.1007/978-1-4471-5779-3 ↗
48. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., and Narayanan, S. (2004, January 13–15). Analysis of emotion recognition using facial expressions, speech and multimodal information. Proceedings of the International Conference on Multimodal Interfaces, State College, PA, USA.10.1145/1027933.1027968 ↗
49. Recognizing emotion in speech
50. A comparative performance study of several pitch detection algorithms10.1109/TASSP.1976.1162846 ↗
51. Speech emotion recognition using Fourier parameters10.1109/TAFFC.2015.2392101 ↗
52. An ongoing review of speech emotion recognition10.1016/j.neucom.2023.01.002 ↗
53. Self supervised adversarial domain adaptation for cross-corpus and cross-language speech emotion recognition10.1109/TAFFC.2022.3167013 ↗
54. Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M.A., Schuller, B., and Zafeiriou, S. (2016, January 20–25). Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472669 ↗
55. Yenigalla, P., Kumar, A., Tripathi, S., Singh, C., Kar, S., and Vepa, J. (2018, January 2–6). Speech Emotion Recognition Using Spectrogram & Phoneme Embedding. Proceedings of the 19th annual conference of the International Speech Communication Association Hyderabad, Hyderabad, India.10.21437/Interspeech.2018-1811 ↗
56. Pepino, L., Riera, P., and Ferrer, L. (2021). Emotion recognition from speech using wav2vec 2.0 embeddings. arXiv, 3400–3404.10.21437/Interspeech.2021-703 ↗
57. Schuller, B., Rigoll, G., and Lang, M. (2003, January 6–9). Hidden Markov model-based speech emotion recognition. Proceedings of the 2003 International Conference on Multimedia and Expo. ICME ’03. Proceedings (Cat. No.03TH8698), Baltimore, MD, USA.10.1109/ICME.2003.1220939 ↗
58. Lin, Y.L., and Wei, G. (2005). Speech emotion recognition based on HMM and SVM. Int. Conf. Mach. Learn. Cybern., 4898–4901.10.1109/ICMLC.2005.1527805 ↗
59. Schölkopf, B., and Smola, A. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press.10.7551/mitpress/4175.001.0001 ↗
60. Vapnik, V. (2013). The Nature of Statistical Learning Theory, Springer Science & Business Media.
61. Cross-corpus speech emotion recognition based on domain-adaptive least-squares regression10.1109/LSP.2016.2537926 ↗
62. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing10.1109/TAFFC.2015.2457417 ↗
63. Eyben, F., Wöllmer, M., and Schuller, B. (2010, January 29). OpenSMILE: The Munich versatile and fast open-source audio feature extractor. Proceedings of the ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1874246 ↗
64. Leem, S.G., Fulford, D., Onnela, J.P., Gard, D., and Busso, C. (2022, January 23–27). Not all features are equal: Selection of robust features for speech emotion recognition in noisy environments. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747705 ↗
65. Automatic versus human speaker verification: The case of voice mimicry10.1016/j.specom.2015.05.002 ↗
66. Pappagari, R., Villalba, J., Żelasko, P., Moro-Velazquez, L., and Dehak, N. (2021, January 6–11). Copypaste: An augmentation method for speech emotion recognition. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9415077 ↗
67. Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., and Tarokh, V. (2020, January 4–8). Speech emotion recognition with dual-sequence LSTM architecture. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054629 ↗
68. A concordance correlation coefficient to evaluate reproducibility10.2307/2532051 ↗
69. Schuller, B., Steidl, S., and Batliner, A. (2009, January 6–10). The Interspeech 2009 emotion challenge. Proceedings of the Interspeech 2009, Brighton, UK.10.21437/Interspeech.2009-103 ↗
70. Kim, J., Englebienne, G., Truong, K.P., and Evers, V. (2017). Towards speech emotion recognition “in the wild” using aggregated corpora and deep multi-task learning. arXiv.10.21437/Interspeech.2017-736 ↗
71. Multi-classifier interactive learning for ambiguous speech emotion recognition10.1109/TASLP.2022.3145287 ↗
72. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings10.1109/TAFFC.2017.2736999 ↗
73. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
74. MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception10.1109/TAFFC.2016.2515617 ↗
75. CREMA-D: Crowd-sourced emotional multimodal actors dataset10.1109/TAFFC.2014.2336244 ↗
76. Costantini, G., Iaderola, I., Paoloni, A., and Todisco, M. (2014, January 26–31). EMOVO corpus: An Italian emotional speech database. Proceedings of the International Conference on Language Resources and Evaluation, Reykjavik, Iceland.
77. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., and Weiss, B. (2005, January 4–8). A database of German emotional speech. Proceedings of the Interspeech 2005, Lisbon, Portugal.10.21437/Interspeech.2005-446 ↗
78. Martin, O., Kotsia, I., Macq, B., and Pitas, I. (2016, January 20). The eNTERFACE’05 audio-visual emotion database. Proceedings of the International Conference on Data Engineering Workshops, Helsinki, Finland.
79. Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.10.1371/journal.pone.0196391 ↗
80. Grimm, M., Kroschel, K., and Narayanan, S. (2008, January 23–26). The Vera am Mittag German audio-visual emotional speech database. Proceedings of the IEEE International Conference on Multimedia and Expo, Hannover, Germany.10.1109/ICME.2008.4607572 ↗
81. Wu, T., Yang, Y., Wu, Z., and Li, D. (2006, January 28–30). MASC: A speech corpus in Mandarin for emotion analysis and affective speaker recognition. Proceedings of the IEEE Odyssey—The Speaker and Language Recognition Workshop, San Juan, Puerto Rico.10.1109/ODYSSEY.2006.248084 ↗
82. Tickle, A. (2000, January 5–7). English and Japanese speakers’ emotion vocalisation and recognition: A comparison highlighting vowel quality. Proceedings of the ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, Northern Ireland, UK.
83. Batliner, A., Steidl, S., and Nöth, E. (2008, January 26). Releasing a thoroughly annotated and processed spontaneous emotional database: The FAU Aibo Emotion Corpus. Proceedings of the Satellite Workshop of LREC 2008 on Corpora for Research on Emotion and Affect, Marrakech, Morocco.
84. EmoMatchSpanishDB: Study of speech emotion recognition machine learning models in a new Spanish elicited database
85. Makarova, V., and Petrushin, V.A. (2002, January 16–20). RUSLANA: A database of Russian emotional utterances. Proceedings of the 7th International Conference on Spoken Language Processing, Denver, CL, USA.10.21437/ICSLP.2002-560 ↗
86. Parry, J., Palaz, D., Clarke, G., Lecomte, P., Mead, R., Berger, M., and Hofer, G. (2019, January 15–19). Analysis of Deep Learning Architectures for Cross-Corpus Speech Emotion Recognition. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-2753 ↗
87. Cross-corpus acoustic emotion recognition: Variances and strategies10.1109/T-AFFC.2010.8 ↗
88. Cai, X., Yuan, J., Zheng, R., Huang, L., and Church, K. (September, January 30). Speech emotion recognition with multi-task learning. Proceedings of the Interspeech 2021, Brno, Czech Republic.10.21437/Interspeech.2021-1852 ↗
89. Chen, L.W., and Rudnicky, A. (2023, January 4–10). Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095036 ↗
90. Dawn of the transformer era in speech emotion recognition: Closing the valence gap10.1109/TPAMI.2023.3263585 ↗
91. Feng, T., and Narayanan, S. (2023, January 10–13). PEFT-SER: On the use of parameter efficient transfer learning approaches for speech emotion recognition using pre-trained speech models. Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Cambridge, MA, USA.10.1109/ACII59096.2023.10388152 ↗
92. Emonet: A transfer learning framework for multi-corpus speech emotion recognition10.1109/TAFFC.2021.3135152 ↗
93. He, S., Zheng, X., Zeng, D., Luo, C., and Zhang, Z. (2016). Exploring entrainment patterns of human emotion in social media. PLoS ONE, 11.10.1371/journal.pone.0150630 ↗
94. Han, K., Yu, D., and Tashev, I. (2014, January 14–18). Speech emotion recognition using deep neural network and extreme learning machine. Proceedings of the Interspeech 2014, Singapore, Singapore.10.21437/Interspeech.2014-57 ↗
95. Satt, A., Rozenberg, S., and Hoory, R. (2017, January 20–24). Efficient emotion recognition from speech using deep learning on spectrograms. Proceedings of the Interspeech 2017, Stockholm, Sweden.10.21437/Interspeech.2017-200 ↗
96. Speech emotion recognition: Features and classification models10.1016/j.dsp.2012.05.007 ↗
97. Evaluating deep learning architectures for speech emotion recognition10.1016/j.neunet.2017.02.013 ↗
98. Bartz, C., Herold, T., Yang, H., and Meinel, C. (2017, January 4–9). Language identification using deep convolutional recurrent neural networks. Proceedings of the Neural Information Processing Conference, Long Beach, CA, USA.10.1007/978-3-319-70136-3_93 ↗
99. Issa, D., Demirci, M.F., and Yazici, A. (2020). Speech emotion recognition with deep convolutional neural networks. Biomed. Signal Process. Control, 59.10.1016/j.bspc.2020.101894 ↗
100. Aftab, A., Morsali, A., Ghaemmaghami, S., and Champagne, B. (2022, January 23–27). LIGHT-SERNET: A lightweight fully convolutional neural network for speech emotion recognition. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746679 ↗