A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis
Source abstract
Abstract
Multimodal sentiment analysis faces a number of challenges, including modality missing, modality heterogeneity gap, incomplete datasets, etc. Previous studies usually adopt schemes like meta-learning or multi-layer structures. Nevertheless, these methods lack interpretability for the interaction between modalities. In this paper, we constructed a new dataset, SM-MSD, for sentiment analysis in social media (SAS) that differs significantly from conventional corpora, comprising 10K instances of diverse data from Twitter, encompassing text, emoticons, emojis, and text embedded in images. This dataset aims to reflect authentic social scenarios and various emotional expressions, and provides a meaningful and challenging evaluation benchmark for multimodal sentiment analysis in specific contexts. Furthermore, we propose a multi-task framework based on heterogeneous graph neural networks (H-GNNs) and contrastive learning. For the first time, heterogeneous graph neural networks are applied to multimodal sentiment analysis tasks. In the case of additional labeling data, it guides the emotion prediction of the missing mode. We conduct extensive experiments on multiple datasets to verify the effectiveness of the proposed scheme. Experimental results demonstrate that our proposed scheme surpasses state-of-the-art methods by 1.7% and 0 in accuracy and 1.54% and 4.9% in F1-score on the MOSI and MOSEI datasets, respectively, and exhibits robustness to modality missing scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Advanced Text Analysis Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Grid Hebei Information and Telecommunication Branch, Shijiazhuang 050051, China
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876, China
Funding
Science and Technology Project of State Grid Hebei Electric Power Co., Ltd.
References (55)
Show source-provided references
1. Chandrasekaran, G., Nguyen, T.N., and Hemanth D, J. (2021). Multimodal sentimental analysis for social media applications: A comprehensive review. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 11.10.1002/widm.1415 ↗
2. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis10.1016/j.inffus.2020.04.003 ↗
3. You, Q., Luo, J., Jin, H., and Yang, J. (2016, January 22–25). Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia. Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, Francisco, CA, USA.10.1145/2835776.2835779 ↗
4. Kaur, R., and Kautish, S. (2022). Multimodal sentiment analysis: A survey and comparison. Research Anthology on Implementing Sentiment Analysis Across Multiple Disciplines, IGI Global.10.4018/978-1-6684-6303-1.ch098 ↗
5. Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions10.1016/j.inffus.2022.09.025 ↗
6. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages10.1109/MIS.2016.94 ↗
7. Gurevych, I., and Miyao, Y. (2018, January 15–20). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.
8. Han, W., Chen, H., Gelbukh, A., Zadeh, A., Morency, L.P., and Poria, S. (2021, January 18–22). Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis. Proceedings of the 2021 International Conference on Multimodal Interaction, Montreal, QC, Canada.10.1145/3462244.3479919 ↗
9. Jiang, M., and Ji, S. (2022). Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis. arXiv.
10. Palmer, M., Hwa, R., and Riedel, S. (2017, January 7–11). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.
11. Hazarika, D., Zimmermann, R., and Poria, S. (2020, January 12–16). Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. Proceedings of the 28th ACM International Conference on Multimedia, Virtual.
12. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv.10.18653/v1/P18-1209 ↗
13. Tsai, Y.H.H., Liang, P.P., Zadeh, A., Morency, L.P., and Salakhutdinov, R. (2018). Learning factorized multimodal representations. arXiv.
14. Yang, D., Huang, S., Kuang, H., Du, Y., and Zhang, L. (2022, January 10–14). Disentangled Representation Learning for Multimodal Emotion Recognition. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Portugal.10.1145/3503161.3547754 ↗
15. Sun, Z., Sarma, P., Sethares, W., and Liang, Y. (2020, January 7–12). Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6431 ↗
16. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
17. Kipf, T.N., and Welling, M. (2016). Variational graph auto-encoders. arXiv.
18. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv.
19. Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks?. arXiv.
20. Graph transformer networks
21. Wang, X., Ji, H., Shi, C., Wang, B., Ye, Y., Cui, P., and Yu, P.S. (2019, January 13–17). Heterogeneous graph attention network. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313562 ↗
22. Korhonen, A., Traum, D., and Màrquez, L. (August, January 28). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
23. Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., and Mariani, J. (2022, January 20–25). Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers. Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France.
24. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
25. Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., and Zimmermann, R. (November, January 31). Icon: Interactive conversational memory network for multimodal emotion detection. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1280 ↗
26. Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., and Zhao, J. (2022, January 14–18). M3Care: Learning with Missing Modalities in Multimodal Healthcare Data. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, NY, USA.10.1145/3534678.3539388 ↗
27. Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions10.1016/j.inffus.2021.12.003 ↗
28. Sun, W., Ma, F., Li, Y., Huang, S.L., Ni, S., and Zhang, L. (2021, January 6–11). Semi-supervised multimodal image translation for missing modality imputation. Proceedings of the ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413461 ↗
29. Multimodal machine learning: A survey and taxonomy
30. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (July, January 28). Multimodal deep learning. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA.
31. Tran, L., Liu, X., Zhou, J., and Jin, R. (2017, January 21–26). Missing modalities imputation via cascaded residual autoencoder. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.528 ↗
32. Zeng, Y., Li, Z., Tang, Z., Chen, Z., and Ma, H. (2023). Heterogeneous graph convolution based on In-domain Self-supervision for Multimodal Sentiment Analysis. Expert Syst. Appl., 213.10.1016/j.eswa.2022.119240 ↗
33. Lu, G., Li, J., and Wei, J. (2022). Aspect sentiment analysis with heterogeneous graph neural networks. Inf. Process. Manag., 59.10.1016/j.ipm.2022.102953 ↗
34. Linmei, H., Yang, T., Shi, C., Ji, H., and Li, X. (2019, January 3–7). Heterogeneous graph attention networks for semi-supervised short text classification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1488 ↗
35. Hamilton, W., Ying, Z., and Leskovec, J. (2017, January 4–9). GraphSAGE: Inductive Representation Learning on Large Graphs. Proceedings of the 2017 Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.10.1109/BigData.2017.8257903 ↗
36. Heterogeneous information network embedding for recommendation10.1109/TKDE.2018.2833443 ↗
37. Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2017). Graph Attention Networks. Stat, 1050.
38. Zhang, C., Song, D., Huang, C., Swami, A., and Chawla, N.V. (2019, January 4–8). Heterogeneous Graph Neural Network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3330961 ↗
39. Yu, L., Shen, J., Li, J., and Lerer, A. (2020). Scalable Graph Neural Networks for Heterogeneous Graphs. arXiv.
40. Multimodal learning with graphs10.1038/s42256-023-00624-6 ↗
41. Gao, D., Li, K., Wang, R., Shan, S., and Chen, X. (2020, January 14–19). Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01276 ↗
42. A comprehensive survey on graph neural networks10.1109/TNNLS.2020.2978386 ↗
43. Imran, M., Yin, H., Chen, T., Huang, Z., and Zheng, K. (2022). DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks. arXiv.10.1109/TKDE.2022.3141951 ↗
44. Heterogeneous graph neural networks analysis: A survey of techniques, evaluations and applications10.1007/s10462-022-10375-2 ↗
45. Yang, J., Wang, Y., Yi, R., Zhu, Y., Rehman, A., Zadeh, A., Poria, S., and Morency, L.P. (2020). Mtgat: Multimodal temporal graph attention networks for unaligned human multimodal language sequences. arXiv.10.18653/v1/2021.naacl-main.79 ↗
46. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv.10.18653/v1/P19-1050 ↗
47. Zadeh, A., Liang, P.P., Poria, S., Vij, P., Cambria, E., and Morency, L.P. (2018, January 2–7). Multi-attention recurrent network for human communication comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12024 ↗
48. Brinzea, R., Khaertdinov, B., and Asteriadis, S. (2022, January 18–23). Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition. Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy.10.1109/IJCNN55064.2022.9892522 ↗
49. Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., and Faieta, B. (2021, January 19–25). Multimodal Contrastive Training for Visual Representation Learning. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.10.1109/CVPR46437.2021.00692 ↗
50. Contrastive learning enables rapid mapping to multimodal single-cell atlas of multimillion scale10.1038/s42256-022-00518-z ↗
51. Liu, W., Qiu, J.L., Zheng, W.L., and Lu, B.L. (2019). Multimodal emotion recognition using deep canonical correlation analysis. arXiv.
52. Padi, S., Sadjadi, S.O., Manocha, D., and Sriram, R.D. (2022). Multimodal emotion recognition using transfer learning from speaker recognition and bert-based models. arXiv.10.21437/Odyssey.2022-57 ↗
53. Liao, W., Zeng, B., Liu, J., Wei, P., Cheng, X., and Zhang, W. (2021). Multi-level graph neural network for text sentiment analysis. Comput. Electr. Eng., 92.10.1016/j.compeleceng.2021.107096 ↗
54. Yang, X., Feng, S., Zhang, Y., and Wang, D. (2021, January 1–6). Multimodal sentiment detection based on multi-channel graph neural networks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual.10.18653/v1/2021.acl-long.28 ↗
55. Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep Learning: The MIT Press, 2016, 800 pp, ISBN: 026203561810.1007/s10710-017-9314-z ↗