Multi-Modal Sentiment Analysis Based on Interactive Attention Mechanism
Source abstract
Abstract
In recent years, multi-modal sentiment analysis has become more and more popular in the field of natural language processing. Multi-modal sentiment analysis mainly concentrates on text, image and audio information. Previous work based on BERT utilizes only text representation to fine-tune BERT, while ignoring the importance of nonverbal information. Most current research methods are fine-tuning models based on BERT that do not optimize BERT’s internal structure. Therefore, in this paper, we propose an optimized BERT model that is composed of three modules: the Hierarchical Multi-head Self Attention module realizes the hierarchical extraction process of the features; the Gate Channel module replaces BERT’s original Feed-Forward layer to realize information filtering; the tensor fusion model based on self-attention mechanism utilized to implement the fusion process of different modal features. In CMU-MOSI, a public mult-imodal sentiment analysis dataset, the accuracy and F1-Score were improved by 0.44% and 0.46% compared with the original BERT model using custom fusion. Compared with traditional models, such as LSTM and Transformer, they are improved to a certain extent.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Advanced Text Analysis Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Hubei University of Technology, Wuhan 430068, China
Funding
National Natural Science Foundation of China · Hubei Province Science and Technology Support Project · The Fundamental Research Funds for the Research Fund of Hubei University of Technology
References (23)
Show source-provided references
1. Machine Learning and Event-Based Software Testing: Classifiers for Identifying Infeasible GUI Event Sequences10.1016/B978-0-12-396535-6.00004-1 ↗
2. Word-level sentiment analysis with reinforcement learning10.1088/1757-899X/490/6/062063 ↗
3. Chen, M., Wang, S., Liang, P.P., Baltrušaitis, T., Zadeh, A., and Morency, L.-P. (2017, January 13–17). Multi-modal sentiment analysis with word-level fusion and reinforcement learning. Proceedings of the 19th ACM International Conference on Multi-Modal Interaction, Glasgow, UK.10.1145/3136755.3136801 ↗
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv.
6. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
7. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv.10.18653/v1/D17-1115 ↗
8. HSI-BERT: Hyperspectral Image Classification Using the Bidirectional Encoder Representation From Transformers10.1109/TGRS.2019.2934760 ↗
9. Zail, C., Huang, K., Wu, L., Zhong, Z., and Jiao, Z. (2022). Relational Graph Convolutional Network for Text-Mining-Based Accident Causal Classification. Appl. Sci., 12.10.3390/app12052482 ↗
10. Zhao, S., Zhang, T., Hu, M., Chang, W., and You, F. (2022). AP-BERT: Enhanced pre-trained model through average pooling. Appl. Intell.10.1007/s10489-022-03190-3 ↗
11. MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis10.1109/LSP.2021.3139856 ↗
12. Zhu, X., Zhu, Y., Zhang, L., and Chen, Y. (2022). A BERT-based multi-semantic learning model with aspect-aware enhancement for aspect polarity classification. Appl. Intell.10.1007/s10489-022-03702-1 ↗
13. Morency, L.-P., Mihalcea, R., and Doshi, P. (2011, January 14–18). Towards multi-modal sentiment analysis: Harvesting opinions from the web. Proceedings of the 13th International Conference on Multi-Modal Interfaces, Alicante, Spain.10.1145/2070481.2070509 ↗
14. Wang, H., Meghawat, A., Morency, L.-P., and Xing, E.P. (2017, January 10–14). Select-additive learning: Improving generalization in multi-modal sentiment analysis. Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019301 ↗
15. Kumar, A., and Vepa, J. (2020, January 4–8). Gated mechanism for attention based multi modal sentiment analysis. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053012 ↗
16. Arjmand, M., Dousti, M.J., and Moradi, H. (2021). Teasel: A transformer-based speech-prefixed language model. arXiv.
17. Multi-layer attention based cnn fortarget-dependent sentiment classification10.1007/s11063-019-10017-9 ↗
18. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
19. Wang, A., and Cho, K. (2019). BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. arXiv.
20. Tsai, Y.-H.H., Liang, P.P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R. (2018). Learning factorized multi-modal representations. arXiv.
21. Liang, P.P., Liu, Z., Zadeh, A., and Morency, L.-P. (2018). Multi-modal language analysis with recurrent multistage fusion. arXiv.10.18653/v1/D18-1014 ↗
22. Pham, H., Liang, P.P., Manzini, T., Morency, L.-P., and Póczos, B. (February, January 27). Found in translation: Learning robust joint representations by cyclic translations between modalities. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
23. Tsai, Y.-H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., and Salakhutdinov, R. (August, January 28). Multi-modal transformer for unaligned multi-modal language sequences. Proceedings of the Association for Computational Linguistics Meeting, Florence, Italy.