Multimodal Emotion Recognition Based on Graph Neural Networks
Source abstract
Abstract
Emotion recognition remains a challenging task in human–computer interaction. With advancements in multimodal computing, multimodal emotion recognition has become increasingly important and significant. To address the existing limitations in multimodal fusion efficiency, emotional–semantic association mining, and long-range context modeling, we propose an innovative graph neural network (GNN)-based framework. Our methodology integrates three key components: (1) a hierarchical sequential fusion (HSF) multimodal integration approach, (2) a sentiment–emotion enhanced joint learning framework, and (3) a context-similarity dual-layer graph architecture (CS-BiGraph). The experimental results demonstrate that our method achieves 69.1% accuracy on the IEMOCAP dataset, establishing a new state-of-the-art performance. For future work, we will explore robust extensions of our framework under real-world scenarios with higher noise levels and investigate the integration of emerging modalities for broader applicability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Educational Service Center, Communication University of China, Beijing 100024, China
School of Information and Engineering, Communication University of China, Beijing 100024, China
School of Data and Intelligence, Communication University of China, Beijing 100024, China
References (31)
Show source-provided references
1. Affective computing: Challenges10.1016/S1071-5819(03)00052-1 ↗
2. Zhou, S., Jia, J., Wang, Q., Dong, Y., Yin, Y., and Lei, K. (2018, January 2–7). Inferring Emotion from Conversational Voice Data: A Semi-Supervised Multi-Path Generative Neural Network Approach. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. Number 1.10.1609/aaai.v32i1.11280 ↗
3. Fusing audio, visual and textual clues for sentiment analysis from multimodal content10.1016/j.neucom.2015.01.095 ↗
4. Hu, G., Lin, T.E., Zhao, Y., Lu, G., Wu, Y., and Li, Y. (2022). UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition. arXiv.10.18653/v1/2022.emnlp-main.534 ↗
5. Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (November, January 31). ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
6. Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., and Cambria, E. (February, January 27). DialogueRNN: An Attentive RNN for Emotion Detection in Conversations. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA. Number 1.
7. Ghosal, D., Majumder, N., Poria, S., Chhaya, N., and Gelbukh, A. (2019). DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation. arXiv.10.18653/v1/D19-1015 ↗
8. Zhang, D., Wu, L., Sun, C., Li, S., Zhu, Q., and Zhou, G. (2019, January 10–16). Modeling both Context- and Speaker-Sensitive Dependence for Emotion Detection in Multi-speaker Conversations. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/752 ↗
9. Tsai, Y.H.H., Bai, S., Pu Liang, P., Kolter, J.Z., Morency, L.P., and Salakhutdinov, R. (August, January 28). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
10. A Low-Rank Matching Attention Based Cross-Modal Feature Fusion Method for Conversational Emotion Recognition10.1109/TAFFC.2024.3498443 ↗
11. Multimodal Emotion Recognition with Auxiliary Sentiment Information
12. Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L.P., and Zimmermann, R. (2018, January 1–6). Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, New Orleans, LA, USA.10.18653/v1/N18-1193 ↗
13. Bao, Y., Ma, Q., Wei, L., Zhou, W., and Hu, S. (2022). Speaker-Guided Encoder-Decoder Framework for Emotion Recognition in Conversation. arXiv.10.24963/ijcai.2022/562 ↗
14. Jiao, W., Lyu, M., and King, I. (2020, January 7–12). Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. Number 5.
15. Webber, B., Cohn, T., He, Y., and Liu, Y. (2020, January 16–20). Relation-aware Graph Attention Networks with Relational Position Encodings for Emotion Recognition in Conversations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.
16. Wu, Y., Zhang, S., and Li, P. (2025). Multi-modal emotion recognition in conversation based on prompt learning with text-audio fusion features. Sci. Rep., 15.10.1038/s41598-025-89758-8 ↗
17. Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv.10.18653/v1/D19-1410 ↗
18. Eyben, F., Wöllmer, M., and Schuller, B. (2010, January 25–29). Opensmile: The munich versatile and fast open-source audio feature extractor. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1874246 ↗
19. Baltrušaitis, T., Robinson, P., and Morency, L.P. (2016, January 7–9). OpenFace: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477553 ↗
20. Joshi, A., Bhat, A., Jain, A., Singh, A.V., and Modi, A. (2022). COGMEN: COntextualized GNN based Multimodal Emotion recognitioN. arXiv.10.18653/v1/2022.naacl-main.306 ↗
21. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
22. Gurevych, I., and Miyao, Y. (2018, January 15–20). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.
23. Moschitti, A., Pang, B., and Daelemans, W. (2014, January 25–29). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.
24. Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations10.1109/TAI.2024.3445325 ↗
25. Yun, T., Lim, H., Lee, J., and Song, M. (2024). TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation. arXiv.10.18653/v1/2024.naacl-long.5 ↗
26. Barzilay, R., and Kan, M.Y. (August, January 30). Context-Dependent Sentiment Analysis in User-Generated Videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
27. Hu, J., Liu, Y., Zhao, J., and Jin, Q. (2021). MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation. arXiv.10.18653/v1/2021.acl-long.440 ↗
28. Delbrouck, J.B., Tits, N., Brousmiche, M., and Dupont, S. (2020, January 10). A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis. Proceedings of the Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), Online.10.18653/v1/2020.challengehml-1.1 ↗
29. Shenoy, A., and Sardana, A. (2020, January 10). Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation. Proceedings of the Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), Online.10.18653/v1/2020.challengehml-1.3 ↗
30. Zhang, Y., Chen, M., Shen, J., and Wang, C. (March, January 22). Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
31. Peng, C., Chen, K., Shou, L., and Chen, G. (2024, January 20–27). CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i13.29374 ↗