Multimodal Emotion Recognition Model Based on Dynamic Heterogeneous Graph Temporal Network
Source abstract
Abstract
To address the semantic gap and complex feature entanglement inherent in multimodal emotion recognition, we propose the Dynamic Heterogeneous Graph Temporal Network (DHGTN), an end-to-end framework designed to model dynamic cross-modal interactions effectively. Utilizing a robust backbone of Wav2vec 2.0, VideoMAE, and BERT, we introduce a “Shared Private” subspace projection mechanism that explicitly disentangles emotion common features from modality-specific noise through contrastive learning to ensure strict semantic alignment. Furthermore, our collaborative Dynamic Heterogeneous Graph and Transformer module overcomes static fusion limitations by constructing time-varying graphs for instantaneous associations and employing global attention to capture long-range temporal dependencies. Extensive experiments on the IEMOCAP and MELD benchmarks demonstrate that DHGTN significantly outperforms state-of-the-art baselines, achieving weighted F1-scores of 73.86% and 66.87%, respectively, which confirms the method’s effectiveness and robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Mental Health via Writing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Inner Mongolia University, Hohhot 010021, China
References (38)
Show source-provided references
1. Li, Y., Wang, Y., Cui, Z., Fu, G., Zhu, W., and Yang, J. (2023, January 18–22). Decoupled multimodal distilling for emotion recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00641 ↗
2. Wang, Y., Li, Y., Cui, Z., Fu, G., and Yang, J. (2023, January 10–16). Incomplete multimodality-diffused emotion recognition. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
3. Luo, W., Xu, M., Lai, H., Bian, Y., and Li, Y. (2023, January 9–12). Multimodal reconstruct and align net for missing modality problem in sentiment analysis. Proceedings of the 29th International Conference on Multimedia Modeling, Bergen, Norway.10.1007/978-3-031-27818-1_34 ↗
4. Improving targeted multimodal sentiment classification with semantic description of images
5. Aspect Affective Triad Extraction Based on Language Feature Enhancement
6. Modality translation-based multimodal sentiment analysis under uncertain missing modalities10.1016/j.inffus.2023.101973 ↗
7. Li, M., Yang, D., Zhao, X., Wang, S., Wang, Y., Yang, K., Sun, M., Kou, D., Qian, Z., and Zhang, L. (2024, January 17–21). Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01184 ↗
8. Sun, H., Zhao, S., Li, S., Kong, X., Wang, X., Kong, A., Zhou, J., Chen, Y., Zeng, W., and Qin, Y. (2025, January 6–11). Enhancing emotion recognition in incomplete data: A novel cross modal alignment, reconstruction, and refinement framework. Proceedings of the 2025 IEEE International Conference on Acoustics, Speech and Signal Processing, Hyderabad, India.10.1109/ICASSP49660.2025.10889485 ↗
9. Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis10.1109/TAFFC.2022.3172360 ↗
10. Yang, J., Yu, Y., Niu, D., Guo, W., and Xu, Y. (2023, January 9–14). Confede: Contrastive feature decomposition for multimodal sentiment analysis. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada.10.18653/v1/2023.acl-long.421 ↗
11. Ma, F., Zhang, Y., Sun, X., Zhao, Y., Cui, L., Dong, R., and Hao, S. (2023, January 10–14). Multimodal sentiment analysis with preferential fusion and distance-aware contrastive learning. Proceedings of the 2023 IEEE International Conference on Multimedia and Expo (ICME), Brisbane, Australia.10.1109/ICME55011.2023.00237 ↗
12. Multimodal Affection Analysis Based on Intensive Collaborative Attention
13. Tu, G., Xie, T., Liang, B., Wang, H., and Xu, R. (2024, January 20–27). Adaptive graph learning for multimodal conversational emotion detection. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
14. A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning10.1016/j.ipm.2024.103675 ↗
15. Zhang, C., Zhang, Y., and Cheng, B. (2024, January 14–19). RL-EMO: A Reinforcement Learning Framework for Multimodal Emotion Recognition. Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446459 ↗
16. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017, January 7–11). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗
17. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.-P. (2018, January 15–20). Efficient Low-rank Multimodal Fusion with Modality-Specific Factors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1209 ↗
18. Mai, S., Hu, H., and Xing, S. (August, January 28). Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
19. Development of an automated transformer-based text analysis framework for monitoring fire door defects in buildings10.1038/s41598-025-27648-9 ↗
20. Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., and Salakhutdinov, R. (August, January 28). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
21. Rahman, W., Hasan, M.K., Lee, S., Bagher Zadeh, A., Mao, C., Morency, L.-P., and Hoque, E. (2020, January 5–10). Integrating Multimodal Information in Large Pretrained Transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.214 ↗
22. Ou, X., Hong, Z., Li, X., Zhang, J., Mao, C., and Lan, Y. (2020, January 12–16). Multimodal Local-Global Attention Network for Affective Video Content Analysis. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.
23. Graph neural network–driven text classification for fire-door defect inspection in pre-completion construction10.1038/s41598-025-28120-4 ↗
24. Hu, J., Liu, H., Zhao, Z., Zhou, J., Li, S., and Ren, H. (2021, January 20–24). MMGCN: Multi-modal Graph Convolution Network for Sentiment Analysis. Proceedings of the 29th ACM International Conference on Multimedia, Chengdu, China.
25. GA2MIF: Graph Attention and Attention-Based Multimodal Information Fusion for Sentiment Analysis
26. Bi-stream graph learning based multimodal fusion for emotion recognition in conversation10.1016/j.inffus.2024.102272 ↗
27. GCNet: Graph completion network for incomplete multimodal learning in conversation
28. Wei, A., Shou, Y., Meng, T., Yin, N., and Li, K. (2023). DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognition. arXiv.
29. Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis10.1038/s41598-024-54872-6 ↗
30. CFN-ESA: A cross modal fusion network with emotion-shift awareness for dialogue emotion recognition10.1109/TAFFC.2024.3389453 ↗
31. Yao, B., and Shi, W. (2024, January 14–19). Speaker-centric multimodal fusion networks for emotion recognition in conversations. Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447720 ↗
32. Zou, S.H., Huang, X.Y., and Shen, X.D. (November, January 29). Multimodal prompt transformer with hybrid contrastive learning for emotion recognition in conversation. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
33. Zadeh, A., Liang, P.P., Majumder, N., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 2–7). Memory Fusion Network for Multi-view Sequential Learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
34. Hazarika, D., Zimmermann, R., and Poria, S. (2020, January 12–16). MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.
35. TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis10.1016/j.knosys.2023.111346 ↗
36. Hybrid cross modal interaction learning for multimodal sentiment analysis10.1016/j.neucom.2023.127201 ↗
37. TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis10.1016/j.patcog.2022.109259 ↗
38. Dynamic hypergraph convolutional network for multimodal sentiment analysis10.1016/j.neucom.2023.126992 ↗