Cross-Modal Sentiment Analysis of Text and Video Based on Bi-GRU Cyclic Network and Correlation Enhancement
Source abstract
Abstract
Cross-modal sentiment analysis is an emerging research area in natural language processing. The core task of cross-modal fusion lies in cross-modal relationship extraction and joint feature learning. The existing research methods of cross-modal sentiment analysis focus on static text, video, audio, and other modality data but ignore the fact that different modality data are often unaligned in practical applications. There is a long-term time dependence among unaligned data sequences, and it is difficult to explore the interaction between different modalities. The paper proposes a sentiment analysis model (UA-BFET) based on feature enhancement technology in unaligned data scenarios, which can perform sentiment analysis on unaligned text and video modality data in social media. Firstly, the model adds a cyclic memory enhancement network across time steps. Then, the obtained cross-modal fusion features with interaction are applied to the unimodal feature extraction process of the next time step in the Bi-directional Gated Recurrent Unit (Bi-GRU) so that the progressively enhanced unimodal features and cross-modal fusion features continuously complement each other. Secondly, the extracted unimodal text and video features taken jointly from the enhanced cross-modal fusion features are subjected to canonical correlation analysis (CCA) and input into the fully connected layer and Softmax function for sentiment analysis. Through experiments executed on unaligned public datasets MOSI and MOSEI, the UA-BFET model has achieved or even exceeded the sentiment analysis effect of text, video, and audio modality fusion and has outstanding advantages in solving cross-modal sentiment analysis in unaligned data scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Advanced Computing and Algorithms · Web Data Mining and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Technology, Hebei University of Economics and Business, Shijiazhuang 050061, China
Funding
Scientific Research and Development Program Project of the Hebei University of Economics and Business
References (42)
Show source-provided references
1. A survey on sentiment classification
2. The bigger picture: Combining econometrics with analytics improve forecasts of movie success10.1287/mnsc.2020.3911 ↗
3. O’Connor, B., Balasubramanyan, R., Routledgeb, B.R., and Smith, N.A. (2010, January 23–26). From tweets to polls: Linking text sentiment to public opinion time series. Proceedings of the Fourth International Conference on Weblogs and Social Media, Washington, DC, USA.
4. Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L.-P. (August, January 30). Context-dependent sentiment analysis in user-generated videos. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada.
5. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
6. Gu, Y., Yang, K.N., Fu, S.Y., Chen, S.H., Li, X.Y., and Marsic, I. (2018, January 15–20). Multimodal affective analysis using hierarchical attention strategy with word-level alignment. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1207 ↗
7. Liang, P.P., Liu, Z.Y., Zadeh, A., and Morency, L.-P. (November, January 31). Multimodal Language Analysis with Recurrent Multistage Fusion. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1014 ↗
8. Tsai, Y.-H.H., Liang, P.P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R. (2019, January 6–9). Learning factorized multimodal representations. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
9. Pham, H., Liang, P.P., Manzini, T., Morency, L.-P., and Poczos, B. (February, January 27). Found in translation: Learning robust joint representations by cyclic translations between modalities. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
10. Mai, S.J., Hu, H.F., and Xing, S.L. (August, January 28). Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing. Proceedings of the 57th Conference of the Association for Computational Linguistics, Florence, Italy.
11. Hazarika, D., Zimmermann, R., and Poria, S. (2022, January 12–16). MISA: Modality-invariant and-specific representations for multimodal sentiment analysis. Proceedings of the 28th ACM international conference on multimedia, Seattle, WA, USA.
12. Dai, W.L., Cahyawijaya, S., Bang, Y.J., and Fung, P. (2021). Weakly-supervised Multi-task Learning for Multimodal Affect Recognition. arXiv.
13. Wang, Y.S., Shen, Y., Liu, Z., Liang, P.P., and Zadeh, A. (February, January 27). Words can shift: Dynamically adjusting word representations using nonverbal behaviors. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
14. Tsai, Y.-H.H., Bai, S.J., Liang, P.P., Kolter, J.Z., Morency, L.-P., and Salakhutdinov, R. (August, January 28). Multimodal transformer for unaligned multimodal language sequences. Proceedings of the 57th Conference of the Association for Computational Linguistics, Florence, Italy.
15. Fu, Z.W., Liu, F., Wang, H.Y., Shen, S.Y., Zhang, J.H., Qi, J.Y., Fu, X.L., and Zhou, A.M. (2021). LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences. arXiv.
16. Zhang, R., Xue, C.G., Qi, Q.F., Lin, L.Y., Zhang, J., and Zhang, L. (2023). Bimodal Fusion Network with Multi-Head Attention for Multimodal Sentiment Analysis. Appl. Sci., 13.10.3390/app13031915 ↗
17. A unimodal reinforced transformer with time squeeze fusion for multimodal sentiment analysis10.1109/LSP.2021.3078074 ↗
18. Lv, F.M., Chen, X., Huang, Y.Y., Duan, L.X., and Lin, G.S. (2021, January 19–25). Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00258 ↗
19. Verbal aggression detection on Twitter comments: Convolutional neural network for short-text sentiment analysis10.1007/s00521-018-3442-0 ↗
20. Learning multi-level deep representations for image emotion classification10.1007/s11063-019-10033-9 ↗
21. Hearing lips and seeing voices10.1038/264746a0 ↗
22. Zadeh, A., Chen, M.H., Poria, S., Cambria, E., and Morency, L.-P. (2017, January 9–11). Tensor fusion network for multimodal sentiment analysis analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗
23. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.-P. (2018, January 15–20). Efficient Low-rank Multimodal Fusion With Modality-Specific Factors. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1209 ↗
24. Yu, W.M., Xu, H., Yuan, Z.Q., and Wu, J.L. (2021, January 2–9). Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.10.1609/aaai.v35i12.17289 ↗
25. Multimodal sentiment analysis based on attention neural network
26. Visual-textual sentiment analysis method based on multi-level spatial attention
27. Sentiment analysis of online users based on multimodal co-attention
28. You, Q.Z., Luo, J.B., Jin, H.L., and Yang, J.C. (2015, January 25–30). Robust image sentiment analysis using progressively trained and domain transferred deep networks. Proceedings of the AAAI conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9179 ↗
29. Multimedia sentiment analysis based on convolutional neural network
30. A cross-modal social media sentiment analysis method based on the fusion of image and text
31. Image-text sentiment analysis based on multi-layer cross-modal attention fusion
32. Speaker identification on the SCOTUS corpus10.1121/1.2935783 ↗
33. Zeng, Z.H., Tu, J.L., Pianfetti, B., Liu, M., Zhang, T., Zhang, Z.Q., Huang, T.S., and Levinson, S.E. (2005, January 20–26). Audio-visual affect recognition through multi-stream fused HMM for HCI. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA.
34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Advances in neural information processing systems, Long Beach, CA, USA.
35. Sahay, S., Okur, E., Kumar, S.H., and Nachman, L. (2020). Low Rank Fusion based Transformers for Multimodal Sequences. arXiv.10.18653/v1/2020.challengehml-1.4 ↗
36. The Weighted Cross-Modal Attention Mechanism With Sentiment Prediction Auxiliary Task for Multimodal Sentiment Analysis10.1109/TASLP.2022.3192728 ↗
37. Hierarchical self-adaptation network for multimodal named entity recognition in social media10.1016/j.neucom.2021.01.060 ↗
38. Text sentiment analysis based on fusion of attention mechnism and BiGRU
39. Guo, X.B., Kong, W.-K.A., and Kot, A.C. Deep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation. IEEE Trans. Multimed., 2022.10.1109/TMM.2022.3142448 ↗
40. Han, W., Chen, H., and Poria, S. (2021, January 7–11). Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Virtual Event / Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.723 ↗
41. Zhao, Z.P., and Wang, K. (2022, January 11–15). Unaligned Multimodal Sequences for Depression Assessment From Speech. Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, Scotland, UK.10.1109/EMBC48229.2022.9871556 ↗
42. Su, L., Hu, C.Q., Li, G.F., and Cao, D.P. (2020). MSAF: Multimodal split attention fusion. arXiv.