Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures
Source abstract
Abstract
Emotion detection holds significant importance in facilitating human–computer interaction, enhancing the depth of engagement. By integrating this capability, we pave the way for forthcoming AI technologies to possess a blend of cognitive and emotional understanding, bridging the divide between machine functionality and human emotional complexity. This progress has the potential to reshape how machines perceive and respond to human emotions, ushering in an era of empathetic and intuitive artificial systems. The primary research challenge involves developing models that can accurately interpret and analyze emotions from both auditory and textual data, whereby auditory data require optimizing CNNs to detect subtle and intense emotional fluctuations in speech, and textual data necessitate access to large, diverse datasets to effectively capture nuanced emotional cues in written language. This paper introduces a novel approach to multimodal emotion recognition, seamlessly integrating speech and text modalities to accurately infer emotional states. Employing CNNs, we meticulously analyze speech using Mel spectrograms, while a BERT-based model processes the textual component, leveraging its bidirectional layers to enable profound semantic comprehension. The outputs from both modalities are combined using an attention-based fusion mechanism that optimally weighs their contributions. The proposed method here undergoes meticulous testing on two distinct datasets: Carnegie Mellon University’s Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset and the Multimodal Emotion Lines Dataset (MELD). The results demonstrate superior efficacy compared to existing frameworks, achieving an accuracy of 88.4% and an F1-score of 87.9% on the CMU-MOSEI dataset, and a notable weighted accuracy (WA) of 67.81% and a weighted F1 (WF1) score of 66.32% on the MELD dataset. This comprehensive system offers precise emotion detection and introduces several significant advancements in the field.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Gachon University, Seongnam 1342, Republic of Korea
Department of Finance and Digital Economics, Tashkent State University of Economics, Tashkent 100066, Uzbekistan
Funding
Korea Agency for Technology and Standards
References (45)
Show source-provided references
1. Sun, Z., Sarma, P., Sethares, W., and Liang, Y. (2020, January 7–12). Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6431 ↗
2. Pepino, L., Riera, P., Ferrer, L., and Gravano, A. (2020, January 4–8). Fusion approaches for emotion recognition from speech using acoustic and text-based features. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054709 ↗
3. Improved speech emotion recognition focusing on high-level data representations and swift feature extraction calculation
4. Text-Based Emotion Recognition Using Deep Learning Approach10.1155/2022/2645381 ↗
5. Zygadło, A., Kozłowski, M., and Janicki, A. (2021). Text-Based Emotion Recognition in English and Polish for Therapeutic Chatbot. Appl. Sci., 11.10.3390/app112110146 ↗
6. Extended deep neural network for facial emotion recognition10.1016/j.patrec.2019.01.008 ↗
7. An efficient deep learning technique for facial emotion recognition10.1007/s11042-021-11298-w ↗
8. Makhmudov, F., Kutlimuratov, A., Akhmedov, F., Abdallah, M.S., and Cho, Y.-I. (2022). Modeling Speech Emotion Recognition via Attention-Oriented Parallel CNN Encoders. Electronics, 11.10.3390/electronics11234047 ↗
9. Lightweight Deep Learning Framework for Speech Emotion Recognition10.1109/ACCESS.2023.3297269 ↗
10. Multimodal emotion recognition based on manifold learning and convolution neural network10.1007/s11042-022-13149-8 ↗
11. Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information10.1109/MMUL.2022.3161411 ↗
12. Speech Emotion Based Sentiment Recognition using Deep Neural Networks10.1088/1742-6596/2236/1/012003 ↗
13. Knowledge-augmented face perception: Prospects for the Bayesian brain-framework to align AI and human vision10.1016/j.concog.2022.103301 ↗
14. Adaptive Multimodal Emotion Detection Architecture for Social Robots10.1109/ACCESS.2022.3149214 ↗
15. End-to-end multimodal affect recognition in real-world environments10.1016/j.inffus.2020.10.011 ↗
16. Hu, G., Lin, T.E., Zhao, Y., Lu, G., Wu, Y., and Li, Y. (2022). UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition. arXiv.10.18653/v1/2022.emnlp-main.534 ↗
17. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the NAACL-HLT, Minneapolis, MN, USA.
18. Pre-trained Deep Convolution Neural Network Model with Attention for Speech Emotion Recognition10.3389/fphys.2021.643202 ↗
19. Ilyosov, A., Kutlimuratov, A., and Whangbo, T.-K. (2021). Deep-Sequence–Aware Candidate Generation for e-Learning System. Processes, 9.10.3390/pr9081454 ↗
20. Chu, I.H., Chen, Z., Yu, X., Han, M., Xiao, J., and Chang, P. (2022). Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics.
21. Chen, W., Xing, X., Chen, P., and Xu, X. (2023). Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition. arXiv.10.1109/TAFFC.2024.3369726 ↗
22. Li, H., Kang, Y., Liu, T., Ding, W., and Liu, Z. (2021). Ctal: Pre-training cross-modal transformer for audio-and-language representations. arXiv.10.18653/v1/2021.emnlp-main.323 ↗
23. Chen, W., Xing, X., Xu, X., Yang, J., and Pang, J. (2022, January 23–27). Key-Sparse Transformer for Multimodal Speech Emotion Recognition. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746598 ↗
24. Liu, F., Shen, S.-Y., Fu, Z.-W., Wang, H.-Y., Zhou, A.-M., and Qi, J.-Y. (2022). LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition. Entropy, 24.10.3390/e24071010 ↗
25. GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation10.1016/j.neucom.2023.126427 ↗
26. Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., and Wang, S. (2023). Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG. IEEE Open J. Eng. Med. Biol., 1–8.10.1109/OJEMB.2023.3240280 ↗
27. A Framework to Evaluate Fusion Methods for Multimodal Emotion Recognition10.1109/ACCESS.2023.3240420 ↗
28. A multi-stage dynamical fusion network for multimodal emotion recognition10.1007/s11571-022-09851-w ↗
29. Zhao, J., Dong, W., Shi, L., Qiang, W., Kuang, Z., Xu, D., and An, T. (2022). Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion. Sensors, 22.10.3390/s22155528 ↗
30. QNMF: A quantum neural network based multimodal fusion system for intelligent diagnosis10.1016/j.inffus.2023.101913 ↗
31. Xu, M., Zhang, F., and Khan, S.U. (2020, January 6–8). Improve Accuracy of Speech Emotion Recognition with Attention Head Fusion. Proceedings of the 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC47524.2020.9031207 ↗
32. BrMcFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., and Nieto, O. (2015, January 6–12). librosa: Audio and music signal analysis in python. Proceedings of the Python in Science Conference, Austin, TX, USA.10.25080/Majora-7b98e3ed-003 ↗
33. Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., and Morency, L.P. (2017, January 18–21). Multi-level multiple attentions for contextual multimodal sentiment analysis. Proceedings of the ICDM, New Orleans, LA, USA.10.1109/ICDM.2017.134 ↗
34. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (August, January 28). MELD: A multimodal multi-party dataset for emotion recognition in conversations. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
35. Zadeh, A.A., Liang, P.P., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 15–20). Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.
36. Mamieva, D., Abdusalomov, A.B., Kutlimuratov, A., Muminov, B., and Whangbo, T.K. (2023). Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features. Sensors, 23.10.3390/s23125475 ↗
37. Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P. (2021, January 6–11). Multimodal end-to-end sparse model for emotion recognition. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.417 ↗
38. Multimodal sentiment analysis: Addressing key issues and setting up the baselines10.1109/MIS.2018.2882362 ↗
39. Multimodal Sentiment Analysis Based on Interactive Transformer and Soft Mapping
40. Delbrouck, J.-B., Tits, N., Brousmiche, M., and Dupont, S. (2020, January 10). A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis. Proceedings of the Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), Seattle, WA, USA.10.18653/v1/2020.challengehml-1.1 ↗
41. Enterprise Strategic Management from the Perspective of Business Ecosystem Construction Based on Multimodal Emotion Recognition10.3389/fpsyg.2022.857891 ↗
42. Dutta, S., and Ganapathy, S. (2023). HCAM—Hierarchical Cross Attention Model for Multi-modal Emotion Recognition. arXiv.
43. Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion10.1007/s00521-022-06913-2 ↗
44. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition10.1109/TAFFC.2022.3141237 ↗
45. A survey on deep learning-based non-invasive brain signals: Recent advances and new frontiers10.1088/1741-2552/abc902 ↗