Global Local Fusion Neural Network for Multimodal Sentiment Analysis
Source abstract
Abstract
With the popularity of social networking services, people are increasingly inclined to share their opinions and feelings on social networks, leading to the rapid increase in multimodal posts on various platforms. Therefore, multimodal sentiment analysis has become a crucial research field for exploring users’ emotions. The complex and complementary interactions between images and text greatly heighten the difficulty of sentiment analysis. Previous works conducted rough fusion operations and ignored the study for fine fusion features for the sentiment task, which did not obtain sufficient interactive information for sentiment analysis. This paper proposes a global local fusion neural network (GLFN), which comprehensively considers global and local fusion features, aggregating these features to analyze user sentiment. The model first extracts overall fusion features by attention modules as modality-based global features. Then, coarse-to-fine fusion learning is applied to build local fusion features effectively. Specifically, the cross-modal module is used for rough fusion, and fine-grained fusion is applied to capture the interaction information between objects and words. Finally, we integrate all features to achieve a more reliable prediction. Extensive experimental results, comparisons, and visualization of public datasets demonstrate the effectiveness of the proposed model for multimodal sentiment classification.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Multimodal Machine Learning Applications · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, School of Computing, Tokyo Institute of Technology, 4259 Nagatsuta, Midori-ku, Yokohama-shi 226-8502, Japan
References (56)
Show source-provided references
1. A Survey of Multimodal Sentiment Analysis10.1016/j.imavis.2017.08.003 ↗
2. A Survey of Sentiment Analysis from Social Media Data10.1109/TCSS.2019.2956957 ↗
3. Personalized Travel Recommendation Based on Sentiment-Aware Multimodal Topic Model10.1109/ACCESS.2019.2935155 ↗
4. Localized Twitter Opinion Mining Using Sentiment Analysis10.1186/s40165-015-0016-4 ↗
5. Kauffmann, E., Peral, J., Gil, D., Ferrández, A., Sellers, R., and Mora, H. (2019). Managing Marketing Decision-Making with Sentiment Analysis: An Evaluation of the Main Product Features Using Text Data Mining. Sustainability, 11.10.3390/su11154235 ↗
6. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., and Narayanan, S. (2004, January 13). Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information. Proceedings of the 6th International Conference on Multimodal Interfaces, State College PA USA, New York, NY, USA.10.1145/1027933.1027968 ↗
7. Bengio, Y., and LeCun, Y. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. Conference Track Proceedings.
8. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 1). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Computer Society, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
9. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures10.1162/neco_a_01199 ↗
10. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015, January 7–13). Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.11 ↗
11. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations10.1007/s11263-016-0981-7 ↗
12. Places: A 10 Million Image Database for Scene Recognition10.1109/TPAMI.2017.2723009 ↗
13. Xu, N., and Mao, W. (2017, January 6). MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis. Proceedings of the Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore.10.1145/3132847.3133142 ↗
14. Jawahar, C.V., Li, H., Mori, G., and Schindler, K. (2018, January 2–6). Robust Deep Multi-Modal Learning Based on Gated Information Fusion Network. Proceedings of the Computer Vision—ACCV 2018, Perth, Australia.
15. Deep Learning Based Fusion Approach for Hate Speech Detection10.1109/ACCESS.2020.3009244 ↗
16. Akhtar, M.S., Chauhan, D., Ghosal, D., Poria, S., Ekbal, A., and Bhattacharyya, P. (2109, January 2–7). Multi-Task Learning for Multi-Modal Emotion Recognition and Sentiment Analysis. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.10.18653/v1/N19-1034 ↗
17. Han, W., Chen, H., and Poria, S. (2021, January 7–11). Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.723 ↗
18. Hazarika, D., Zimmermann, R., and Poria, S. (2020, January 12–16). MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis. Proceedings of the 28th ACM International Conference on Multimedia, New York, NY, USA.
19. Han, W., Chen, H., Gelbukh, A., Zadeh, A., Morency, L., and Poria, S. (2021, January 19–22). Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis. Proceedings of the 2021 International Conference on Multimodal Interaction, Montreal, QC, Canada.10.1145/3462244.3479919 ↗
20. Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages10.1109/MIS.2016.94 ↗
21. Bagher Zadeh, A., Liang, P.P., Poria, S., Cambria, E., and Morency, L.-P. (2012, January 8–14). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jeju Island, Korea.
22. Kruk, J., Lubin, J., Sikka, K., Lin, X., Jurafsky, D., and Divakaran, A. (2019, January 3–7). Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1469 ↗
23. Vempala, A., and Preoţiuc-Pietro, D. (August, January 28). Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
24. Henning, C.A., and Ewerth, R. (2019, January 6–9). Estimating the Information Gap between Textual and Visual Representations. Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, Bucharest, Romania.10.1145/3078971.3078991 ↗
25. Characterization and Classification of Semantic Image-Text Relations10.1007/s13735-019-00187-6 ↗
26. Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., and Liu, X. (2016, January 4–6). Sentiment Analysis on Multi-View Social Data. Proceedings of the MultiMedia Modeling, Miami, FL, USA.10.1007/978-3-319-27671-7 ↗
27. Park, S., and Kim, Y. (2016, January 8–10). Building Thesaurus Lexicon Using Dictionary-Based Approach for Sentiment Classification. Proceedings of the 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), Towson, MD, USA.10.1109/SERA.2016.7516126 ↗
28. Pang, B., Lee, L., and Vaithyanathan, S. (2004, January 25–26). Thumbs up? Sentiment Classification Using Machine Learning Techniques. Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), Barcelona, Spain.10.3115/1118693.1118704 ↗
29. Using VADER Sentiment and SVM for Predicting Customer Response Sentiment10.1016/j.eswa.2020.113746 ↗
30. ABCDM: An Attention-Based Bidirectional CNN-RNN Deep Model for Sentiment Analysis10.1016/j.future.2020.08.005 ↗
31. Siersdorfer, S., Minack, E., Deng, F., and Hare, J. (2010, January 25–29). Analyzing and Predicting Sentiment of Images on the Social Web. Proceedings of the 18th ACM international conference on Multimedia, Firenze, Italy.10.1145/1873951.1874060 ↗
32. Borth, D., Ji, R., Chen, T., Breuel, T., and Chang, S.-F. (2013, January 21–25). Large-Scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs. Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain.10.1145/2502081.2502282 ↗
33. Yang, Y., Jia, J., Zhang, S., Wu, B., Chen, Q., Li, J., Xing, C., and Tang, J. (2014, January 27–31). How Do Your Friends on Social Media Disclose Your Emotions?. Proceedings of the AAAI Conference on Artificial Intelligence, Québec, QC, Canada.10.1609/aaai.v28i1.8740 ↗
34. Chen, T., Borth, D., Darrell, T., and Chang, S.-F. (2014). DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks. arXiv.
35. Boosting Image Sentiment Analysis with Visual Attention10.1016/j.neucom.2018.05.104 ↗
36. Visual Sentiment Analysis by Combining Global and Local Information10.1007/s11063-019-10027-7 ↗
37. A Multimodal Feature Learning Approach for Sentiment Analysis of Social Network Multimedia10.1007/s11042-015-2646-x ↗
38. Hu, A., and Flaxman, S. (2018, January 19). Multimodal Sentiment Analysis to Explore the Structure of Emotions. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219853 ↗
39. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). GloVe: Global Vectors for Word Representation. Proceedings of the Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
40. Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12–15). Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0055 ↗
41. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017, January 7–11). Tensor Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗
42. Xu, N., Mao, W., and Chen, G. (2018, January 8–12). A Co-Memory Network for Multimodal Sentiment Analysis. Proceedings of the The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210093 ↗
43. Lauw, H.W., Wong, R.C.-W., Ntoulas, A., Lim, E.-P., Ng, S.-K., and Pan, S.J. (2020, January 11–14). Fusion-Extraction Network for Multimodal Sentiment Analysis. Proceedings of the Advances in Knowledge Discovery and Data Mining, Singapore.10.1007/978-3-030-47426-3 ↗
44. Hu, X., and Yamamura, M. (2022, January 18–20). Two-Stage Attention-Based Fusion Neural Network for Image-Text Sentiment Classification. Proceedings of the 2022 4th International Conference on Image, Video and Signal Processing, Singapore.10.1145/3531232.3531233 ↗
45. Image-Text Multimodal Emotion Classification via Multi-View Attentional Network10.1109/TMM.2020.3035277 ↗
46. Li, Z., Xu, B., Zhu, C., and Zhao, T. (2022). CLMLF: A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection. arXiv.10.18653/v1/2022.findings-naacl.175 ↗
47. Zhu, T., Li, L., Yang, J., Zhao, S., Liu, H., and Qian, J. (2022). Multimodal Sentiment Analysis with Image-Text Interaction Network. IEEE Trans. Multimed., 1.10.1109/TMM.2022.3160060 ↗
48. Bengio, Y., and LeCun, Y. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA. Workshop Track Proceedings.
49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
50. Pre-Trained Models for Natural Language Processing: A Survey10.1007/s11431-020-1647-3 ↗
51. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). VL-BERT: Pre-Training of Generic Visual-Linguistic Representations. arXiv.
52. Tan, H., and Bansal, M. (2019, January 3–7). LXMERT: Learning Cross-Modality Encoder Representations from Transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1514 ↗
53. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 1). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning PMLR, San Francisco, CA, USA.
54. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2022, January 19–24). FLAVA: A Foundational Language and Vision Alignment Model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01519 ↗
55. Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (2022). CoCa: Contrastive Captioners Are Image-Text Foundation Models. arXiv.
56. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.