Multimodal Affective Communication Analysis: Fusing Speech Emotion and Text Sentiment Using Machine Learning
Source abstract
Abstract
Affective communication, encompassing verbal and non-verbal cues, is crucial for understanding human interactions. This study introduces a novel framework for enhancing emotional understanding by fusing speech emotion recognition (SER) and sentiment analysis (SA). We leverage diverse features and both classical and deep learning models, including Gaussian naive Bayes (GNB), support vector machines (SVMs), random forests (RFs), multilayer perceptron (MLP), and a 1D convolutional neural network (1D-CNN), to accurately discern and categorize emotions in speech. We further extract text sentiment from speech-to-text conversion, analyzing it using pre-trained models like bidirectional encoder representations from transformers (BERT), generative pre-trained transformer 2 (GPT-2), and logistic regression (LR). To improve individual model performance for both SER and SA, we employ an extended dynamic Bayesian mixture model (DBMM) ensemble classifier. Our most significant contribution is the development of a novel two-layered DBMM (2L-DBMM) for multimodal fusion. This model effectively integrates speech emotion and text sentiment, enabling the classification of more nuanced, second-level emotional states. Evaluating our framework on the EmoUERJ (Portuguese) and ESD (English) datasets, the extended DBMM achieves accuracy rates of 96% and 98% for SER, 85% and 95% for SA, and 96% and 98% for combined emotion classification using the 2L-DBMM, respectively. Our findings demonstrate the superior performance of the extended DBMM for individual modalities compared to individual classifiers and the 2L-DBMM for merging different modalities, highlighting the value of ensemble methods and multimodal fusion in affective communication analysis. The results underscore the potential of our approach in enhancing emotional understanding with broad applications in fields like mental health assessment, human–robot interaction, and cross-cultural communication.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Physics, Engineering and Computer Science, University of Hertfordshire, Hertfordshire AL10 9AB, UK
AI-Weinberg AI Experts, Tel Aviv 90850, Israel
LABTED and Computer Science Department, State University of Londrina, Londrina 86057-970, Brazil
References (43)
Show source-provided references
1. Facial emotion recognition using deep learning: Review and insights10.1016/j.procs.2020.07.101 ↗
2. Faria, D.R., Vieria, M., Faria, F.C.C., and Premebida, C. (2017, January 28–31). Affective Facial Expressions Recognition for Human-Robot Interaction. Proceedings of the IEEE RO-MAN’17: IEEE International Symposium on Robot and Human Interactive Communication, Lisbon, Portugal.10.1109/ROMAN.2017.8172395 ↗
3. Golzadeh, H., Faria, D.R., Manso, L., Ekart, A., and Buckingham, C. (2018, January 25–27). Emotion Recognition using Spatiotemporal Features from Facial Expression Landmarks. Proceedings of the 9th IEEE International Conference on Intelligent Systems, Madeira, Portugal.10.1109/IS.2018.8710573 ↗
4. Faria, D.R., Vieira, M., and Faria, F.C.C. (2017, January 21–23). Towards the Development of Affective Facial Expression Recognition for Human-Robot Interaction. Proceedings of the ACM PETRA’17: 10th International Conference on Pervasive Technologies Related to Assistive Environments, Island of Rhodes, Greece.10.1145/3056540.3076199 ↗
5. Bird, J.J., Ekart, A., Buckingham, C.D., and Faria, D.R. (2019, January 29–30). Mental Emotional Sentiment Classification with an EEG-based Brain-Machine Interface. Proceedings of the International Conference on Digital Image & Signal Processing (DISP’19), Oxford, UK.
6. Manoharan, G., and Faria, D.R. (2024, January 5–6). Enhanced Mental State Classification using EEG-based Brain-Computer Interface through Deep Learning. Proceedings of the IntelliSys’24: 10th Intelligent Systems Conference, Amsterdam, The Netherlands.10.1007/978-3-031-66431-1_40 ↗
7. Faria, D.R., Premebida, C., and Nunes, U.J. (2014, January 25–29). A Probabilistic Approach for Human Everyday Activities Recognition using Body Motion from RGB-D Images. Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’14), Scotland, UK.10.1109/ROMAN.2014.6926340 ↗
8. Lieskovská, E., Jakubec, M., Jarina, R., and Chmulík, M. (2021). A Review on Speech Emotion Recognition Using Deep Learning and Attention Mechanism. Electronics, 10.10.3390/electronics10101163 ↗
9. Schuller, B.W., Batliner, A., Bergler, C., Pokorny, F.B., Krajewski, J., Cychosz, M., Vollmann, R., Roelen, S.-D., Schnieder, S., and Bergelson, E. (2019). The INTERSPEECH 2019 Computational Paralinguistics Challenge: Styrian Dialects, Continuous Sleepiness, Baby Sounds & Orca Activity. Proc. Interspeech, 2378–2382.10.21437/Interspeech.2019-1122 ↗
10. Costantini, G., Parada-Cabaleiro, E., Casali, D., and Cesarini, V. (2022). The Emotion Probe: On the Universality of Cross-Linguistic and Cross-Gender Speech Emotion Recognition via Machine Learning. Sensors, 22.10.3390/s22072461 ↗
11. Chakhtouna, A., Sekkate, S., and Adib, A. (2022, January 19–21). Speech Emotion Recognition Using Pre-trained and Fine-Tuned Transfer Learning Approaches. Proceedings of the International Conference on Smart City Applications, Sydney, Australia.10.1007/978-3-031-26852-6_35 ↗
12. Self-attention transfer networks for speech emotion recognition10.1016/j.vrih.2020.12.002 ↗
13. Multi-Task Semi-Supervised Adversarial Autoencoding for Speech Emotion Recognition10.1109/TAFFC.2020.2983669 ↗
14. Novais, R., Cardoso, P.J., and Rodrigues, J.M.F. (September, January 31). Emotion classification from speech by an ensemble strategy. Proceedings of the International Conference on Software Development and Technology for Enhancing Accessibility and Fighting Info-Exclusion, Lisboa, Portugal.
15. Lexicon and Deep Learning-Based Approaches in Sentiment Analysis on Short Texts10.4236/jcc.2024.121002 ↗
16. Xu, D., Tian, Z., Lai, R., Kong, X., Tan, Z., and Shi, W. (2020). Deep learning-based emotion analysis of microblog texts. Info. Fusion.10.1016/j.inffus.2020.06.002 ↗
17. Lifelong Text-Audio Sentiment Analysis learning10.1016/j.neunet.2023.02.008 ↗
18. Sentiment analysis on a low-resource language dataset using multimodal representation learning and cross-lingual transfer learning10.1016/j.asoc.2024.111553 ↗
19. Kumar, P., Malik, S., Li, X., and Raman, B. (2022). Hybrid Fusion based Interpretable Multimodal Emotion Recognition with Limited Labelled Data. arXiv.
20. Fusing audio, visual and textual clues for sentiment analysis from multimodal content10.1016/j.neucom.2015.01.095 ↗
21. Atmaja, B.T., and Sasou, A. (2022). Sentiment Analysis and Emotion Recognition from Speech using Universal Speech Representations. Sensors, 22.10.3390/s22176369 ↗
22. Can people feel happy and sad at the same time?10.1037/0022-3514.81.4.684 ↗
23. Beck, A.T. (1967). Depression: Clinical, Experimental and Theoretical Aspects, Harper and Row.
24. American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders, APA. [5th ed.]. (DSM-5).10.1176/appi.books.9780890425596 ↗
25. Hatfield, E., Cacioppo, J.T., and Rapson, R.L. (1994). Emotional Contagion, Cambridge University Press.10.1017/CBO9781139174138 ↗
26. Vaillant, G.E. (1977). Adaptation to Life, Little Brown and Co.
27. Subjective well-being: The science of happiness and a proposal for a national index10.1037/0003-066X.55.1.34 ↗
28. Optimism10.1016/j.cpr.2010.01.006 ↗
29. The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior10.1207/S15327965PLI1104_01 ↗
30. Schneider, K.J. (1996). The Paradoxical Self: Toward an Understanding of our Contradictory Nature, Human Sciences Press.
31. Frijda, N.H. (1986). The Emotions, Cambridge University Press.
32. Human aggression10.1146/annurev.psych.53.100901.135231 ↗
33. Berkowitz, L. (1993). Aggression: Its Causes, Consequences, and Control, McGraw-Hill.
34. Graph Regularization Methods in Soft Detector Fusion10.1109/ACCESS.2023.3344776 ↗
35. Multiclass Alpha Integration of Scores from Multiple Classifiers10.1162/neco_a_01169 ↗
36. Bastos Germano, R.G., Pompeu Tcheou, M., da Rocha Henriques, F., and Pinto Gomes, S. (2021). EmoUERJ: An emotional speech database in Portuguese. Zenodo.
37. Emotional Voice Conversion: Theory, Databases and ESD10.1016/j.specom.2021.11.006 ↗
38. Duret, J., Estève, Y., and Parcollet, T. (2023, January 26–28). Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Data. Proceedings of the 12th ISCA Speech Synthesis Workshop (SSW2023), Grenoble, France.10.21437/SSW.2023-29 ↗
39. A Survey on Transfer Learning10.1109/TKDE.2009.191 ↗
40. Thumbs Up, Thumbs Down: Non-verbal Human-Robot Interaction through Real-time EMG Classification via Inductive and Supervised Transductive Transfer Learning10.1007/s12652-020-01852-z ↗
41. Hussain, M., Bird, J., and Faria, D.R. (2018, January 5–7). A Study on CNN Transfer Learning for Image Classification. Proceedings of the UKCI’18: 18th Annual UK Workshop on Computational Intelligence, Nottingham, UK.
42. Etelis, I., Rosenfeld, A., Weinberg, A.I., and Sarne, D. (2024). Generating Effective Ensembles for Sentiment Analysis. arXiv.
43. Note on the sampling error of the difference between correlated proportions or percentages10.1007/BF02295996 ↗