Deep Multimodal Emotion Recognition on Human Speech: A Review
Source abstract
Abstract
This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Speech and Audio Processing · Infant Health and Development
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Informatics and Telecommunications, National Center for Scientific Research—Demokritos, 15310 Athens, Greece
References (92)
Show source-provided references
1. Morency, L.P., Mihalcea, R., and Doshi, P. (2011, January 14–18). Towards multimodal sentiment analysis: Harvesting opinions from the web. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.10.1145/2070481.2070509 ↗
2. Müller, F.M. (1865). Lectures on the Science of Language: Delivered at the Royal Institution of Great Britain in February, March, April and May, 1863, Charles Scribner.
3. Imaging first impressions: Distinct neural processing of verbal and nonverbal social information10.1016/j.neuroimage.2011.12.046 ↗
4. Functional and anatomical decomposition of face processing: Evidence from prosopagnosia and PET study of normal subjects10.1098/rstb.1992.0007 ↗
5. Neural synchronization during face-to-face communication10.1523/JNEUROSCI.2926-12.2012 ↗
6. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017, January 9–11). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗
7. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (July, January 28). Multimodal Deep Learning. Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA.
8. Pérez-Rosas, V., Mihalcea, R., and Morency, L.P. (2013, January 4–9). Utterance-level multimodal sentiment analysis. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria.
9. Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12–15). Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0055 ↗
10. Wang, H., Meghawat, A., Morency, L.P., and Xing, E.P. (2017, January 10–14). Select-additive learning: Improving generalization in multimodal sentiment analysis. Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019301 ↗
11. Zadeh, A., Zellers, R., Pincus, E., and Morency, L.P. (2016). Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv.
12. Emotion recognition from speech: A review10.1007/s10772-011-9125-1 ↗
13. Lalitha, S., Mudupu, A., Nandyala, B.V., and Munagala, R. (2015, January 10–12). Speech emotion recognition using DWT. Proceedings of the 2015 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Madurai, India.10.1109/ICCIC.2015.7435630 ↗
14. A comparative analysis of classifiers in emotion recognition through acoustic features10.1007/s10772-014-9239-3 ↗
15. Gupta, D., Bansal, P., and Choudhary, K. (2018). The state of the art of feature extraction techniques in speech recognition. Speech and Language Processing for Human-Machine Communications: Proceedings of CSI 2015, Springer.10.1007/978-981-10-6626-9_22 ↗
16. Gupta, K., and Gupta, D. (2016, January 14–15). An analysis on LPC, RASTA and MFCC techniques in Automatic Speech recognition system. Proceedings of the 2016 6th International Conference-Cloud System and Big Data Engineering (Confluence), Noida, India.10.1109/CONFLUENCE.2016.7508170 ↗
17. Voice emotion recognition by cochlear-implanted children and their normally-hearing peers10.1016/j.heares.2014.10.003 ↗
18. Analysis of speech features and personality traits10.1016/j.bspc.2019.01.027 ↗
19. Teager, H., and Teager, S. (1990). Evidence for nonlinear sound production mechanisms in the vocal tract. Speech Production and Speech Modelling, Springer.10.1007/978-94-009-2037-8_10 ↗
20. Kaiser, J.F. (1993, January 27–30). Some useful properties of Teager’s energy operators. Proceedings of the 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, Minneapolis, MN, USA.10.1109/ICASSP.1993.319457 ↗
21. Nonlinear feature based classification of speech under stress10.1109/89.905995 ↗
22. McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., and Nieto, O. (2015, January 6–12). librosa: Audio and music signal analysis in python. Proceedings of the 14th Python in Science Conference, Austin, TX, USA.10.25080/Majora-7b98e3ed-003 ↗
23. Giannakopoulos, T. (2015). pyAudioAnalysis: An open-source python library for audio signal analysis. PLoS ONE, 10.10.1371/journal.pone.0144610 ↗
24. Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. (2014, January 4–9). COVAREP—A collaborative voice analysis repository for speech technologies. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (Icassp), Florence, Italy.10.1109/ICASSP.2014.6853739 ↗
25. Eyben, F., Wöllmer, M., and Schuller, B. (2010, January 25–29). Opensmile: The munich versatile and fast open-source audio feature extractor. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1874246 ↗
26. Learning salient features for speech emotion recognition using convolutional neural networks10.1109/TMM.2014.2360798 ↗
27. Badshah, A.M., Ahmad, J., Rahim, N., and Baik, S.W. (2017, January 13–15). Speech emotion recognition from spectrograms with deep convolutional neural network. Proceedings of the 2017 International Conference on Platform Technology and Service (PlatCon), Busan, Korea.10.1109/PlatCon.2017.7883728 ↗
28. Evaluating deep learning architectures for Speech Emotion Recognition10.1016/j.neunet.2017.02.013 ↗
29. Ravanelli, M., and Bengio, Y. (2018, January 18–21). Speaker recognition from raw waveform with sincnet. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639585 ↗
30. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019). wav2vec: Unsupervised pre-training for speech recognition. arXiv.10.21437/Interspeech.2019-1873 ↗
31. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv.
32. Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv.10.1109/TASLP.2021.3122291 ↗
33. Mikolov, T., Yih, W.T., and Zweig, G. (2013, January 9–14). Linguistic regularities in continuous space word representations. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, GA, USA.
34. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
35. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
36. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
38. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2021, August 27). Improving Language Understanding by Generative Pre-Training. Available online: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
39. Language models are unsupervised multitask learners
40. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
41. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
42. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv.
43. Facial expression and emotion10.1037/0003-066X.48.4.384 ↗
44. Marechal, C., Mikolajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., and Wegrzyn-Wolska, K. (2019). Survey on AI-Based Multimodal Methods for Emotion. High-Performance Modelling and Simulation for Big Data Applications: Selected Results of the COST Action IC1406 cHiPSet, Springer International Publishing.10.1007/978-3-030-16272-6_11 ↗
45. The human face as a dynamic tool for social communication10.1016/j.cub.2015.05.052 ↗
46. Ekman, P., Friesen, W., and Hager, J. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press, Inc.10.1037/t27734-000 ↗
47. Hjortsjö, C. (1969). Man’s Face and Mimic Language, Studentlitteratur.
48. De la Torre, F., Chu, W.S., Xiong, X., Vicente, F., Ding, X., and Cohn, J. (2015, January 4–8). IntraFace. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.10.1109/FG.2015.7163082 ↗
49. iMotions Global (2021, August 27). Facial Expression Analysis Solutions. Available online: https://imotions.com/biosensor/fea-facial-expression-analysis/.
50. Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., and Morency, L.P. (2016, January 12–16). Deep multimodal fusion for persuasiveness prediction. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.10.1145/2993148.2993176 ↗
51. Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L.P. (August, January 30). Context-dependent sentiment analysis in user-generated videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
52. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A.B., and Morency, L.P. (2018, January 15–20). Efficient Low-rank Multimodal Fusion with Modality-Specific Factors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1209 ↗
53. Yoon, S., Byun, S., and Jung, K. (2018, January 18–21). Multimodal speech emotion recognition using audio and text. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639583 ↗
54. Yoon, S., Byun, S., Dey, S., and Jung, K. (2019, January 12–17). Speech emotion recognition using multi-hop attention mechanism. Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683483 ↗
55. Chen, M., and Zhao, X. (2020, January 25–29). A multi-scale fusion framework for bimodal speech emotion recognition. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-3156 ↗
56. Priyasad, D., Fernando, T., Denman, S., Sridharan, S., and Fookes, C. (2020, January 4–8). Attention driven fusion for multi-modal emotion recognition. Proceedings of the ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054441 ↗
57. Liang, P.P., Zadeh, A., and Morency, L.P. (2018, January 16–20). Multimodal local-global ranking fusion for emotion recognition. Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, USA.10.1145/3242969.3243019 ↗
58. Pham, H., Liang, P.P., Manzini, T., Morency, L.P., and Póczos, B. (February, January 27). Found in translation: Learning robust joint representations by cyclic translations between modalities. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
59. Tsai, Y.H.H., Liang, P.P., Zadeh, A., Morency, L.P., and Salakhutdinov, R. (May, January 30). Learning Factorized Multimodal Representations. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
60. Rajagopalan, S.S., Morency, L.P., Baltrusaitis, T., and Goecke, R. (2016, January 11–14). Extending long short-term memory for multi-view structured learning. Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46478-7_21 ↗
61. Zadeh, A., Liang, P.P., Poria, S., Vij, P., Cambria, E., and Morency, L.P. (2018, January 2–7). Multi-attention recurrent network for human communication comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12024 ↗
62. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
63. Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., and Morency, L.P. (2018, January 15–20). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.
64. Liang, P.P., Liu, Z., Zadeh, A.B., and Morency, L.P. (November, January 31). Multimodal Language Analysis with Recurrent Multistage Fusion. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1014 ↗
65. Deep multimodal multilinear fusion with high-order polynomial pooling
66. Mai, S., Hu, H., and Xing, S. (2020, January 7–12). Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i01.5347 ↗
67. Sun, Z., Sarma, P., Sethares, W., and Liang, Y. (2020, January 7–12). Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6431 ↗
68. Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., and Li, X. (2019). Learning Alignment for Multimodal Emotion Recognition from Speech. arXiv.10.21437/Interspeech.2019-3247 ↗
69. Wang, Y., Shen, Y., Liu, Z., Liang, P.P., Zadeh, A., and Morency, L.P. (February, January 27). Words can shift: Dynamically adjusting word representations using nonverbal behaviors. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
70. Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., and Salakhutdinov, R. (2019, January 2–7). Multimodal transformer for unaligned multimodal language sequences. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.10.18653/v1/P19-1656 ↗
71. Yang, J., Wang, Y., Yi, R., Zhu, Y., Rehman, A., Zadeh, A., Poria, S., and Morency, L.P. (2021, January 6–11). MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico.10.18653/v1/2021.naacl-main.79 ↗
72. Speaker-sensitive emotion recognition via ranking: Studies on acted and spontaneous speech10.1016/j.csl.2014.01.003 ↗
73. Basu, S., Chakraborty, J., Bag, A., and Aftabuddin, M. (2017, January 10–11). A review on emotion recognition using speech. Proceedings of the 2017 International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore, India.10.1109/ICICCT.2017.7975169 ↗
74. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
75. Ringeval, F., Sonderegger, A., Sauer, J., and Lalanne, D. (2013, January 22–26). Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553805 ↗
76. Grimm, M., Kroschel, K., and Narayanan, S. (April, January 23). The Vera am Mittag German audio-visual emotional speech database. Proceedings of the 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany.10.1109/ICME.2008.4607572 ↗
77. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild10.1109/TPAMI.2019.2944808 ↗
78. Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.C., Devillers, L., Abrilian, S., and Batliner, A. (2007, January 12–14). The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data. Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction, Lisbon, Portugal.
79. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent10.1109/T-AFFC.2011.20 ↗
80. Collecting large, richly annotated facial-expression databases from movies
81. AFEW-VA database for valence and arousal estimation in-the-wild10.1016/j.imavis.2017.02.001 ↗
82. The MAHNOB Mimicry Database: A database of naturalistic human interactions10.1016/j.patrec.2015.03.005 ↗
83. Siriwardhana, S., Reis, A., Weerasekera, R., and Nanayakkara, S. (2020). Jointly Fine-Tuning “BERT-like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition. arXiv.10.21437/Interspeech.2020-1212 ↗
84. Shenoy, A., and Sardana, A. (2020, January 10). Multilogue-Net: A Context-Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation. Proceedings of the Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), Online.10.18653/v1/2020.challengehml-1.3 ↗
85. Analyzing multimodal sentiment via acoustic-and visual-LSTM with channel-aware temporal convolution network10.1109/TASLP.2021.3068598 ↗
86. Tong, E., Zadeh, A., Jones, C., and Morency, L.P. (August, January 30). Combating human trafficking with multimodal deep models. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
87. Metallinou, A., Lee, S., and Narayanan, S. (2010, January 14–19). Decision level combination of multiple modalities for recognition and analysis of emotional expression. Proceedings of the 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, Dallas, TX, USA.10.1109/ICASSP.2010.5494890 ↗
88. What makes the difference? An empirical comparison of fusion strategies for multimodal language analysis10.1016/j.inffus.2020.09.005 ↗
89. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
90. Autoencoder-based unsupervised domain adaptation for speech emotion recognition10.1109/LSP.2014.2324759 ↗
91. Abdelwahab, M., and Busso, C. (2015, January 19–24). Supervised domain adaptation for emotion recognition from speech. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178934 ↗
92. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗