Dynamic Tuning and Multi-Task Learning-Based Model for Multimodal Sentiment Analysis
Source abstract
Abstract
Multimodal sentiment analysis aims to uncover human affective states by integrating data from multiple sensory sources. However, previous studies have focused on optimizing model architecture, neglecting the impact of objective function settings on model performance. Given this, this study introduces a new framework, DMMSA, which utilizes the intrinsic correlation of sentiment signals and enhances the model’s understanding of complex sentiments. DMMSA incorporates coarse-grained sentiment analysis to reduce task complexity. Meanwhile, it embeds a contrastive learning mechanism within the modality, which decomposes unimodal features into similar and dissimilar ones, thus allowing for the simultaneous consideration of both unimodal and multimodal emotions. We tested DMMSA on the CH-SIMS, MOSI, and MOEI datasets. When only changing the optimization objectives, DMMSA achieved accuracy gains of 3.2%, 1.57%, and 1.95% over the baseline in five-class and seven-class classification tasks. In regression tasks, DMMSA reduced the Mean Absolute Error (MAE) by 1.46%, 1.5%, and 2.8% compared to the baseline.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Advanced Text Analysis Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Information Research Center of Military Science, Beijing 100142, China
Funding
National Natural Science Foundation of China · Natural Science Foundation of Xinjiang, China · Tianshan Talents Cultivation Program—Leadings Talents for Scientific and Technological Innovation
References (40)
Show source-provided references
1. Yang, D., Li, M., Xiao, D., Liu, Y., Yang, K., Chen, Z., Wang, Y., Zhai, P., Li, K., and Zhang, L. (2024). Towards multimodal sentiment analysis debiasing via bias purification. arXiv.10.1007/978-3-031-73636-0_27 ↗
2. Du, J., Jin, J., Zhuang, J., and Zhang, C. (2024). Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis. Sci. Rep., 14.10.1038/s41598-024-54872-6 ↗
3. Modality translation-based multimodal sentiment analysis under uncertain missing modalities10.1016/j.inffus.2023.101973 ↗
4. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis10.1109/TAFFC.2023.3274829 ↗
5. Yang, J., Yu, Y., Niu, D., Guo, W., and Xu, Y. (2023, January 9–14). ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.421 ↗
6. Dynamic hypergraph convolutional network for multimodal sentiment analysis10.1016/j.neucom.2023.126992 ↗
7. Disentanglement Translation Network for multimodal sentiment analysis10.1016/j.inffus.2023.102031 ↗
8. Hu, G., Lin, T.E., Zhao, Y., Lu, G., Wu, Y., and Li, Y. (2022). Unimse: Towards unified multimodal sentiment analysis and emotion recognition. arXiv.10.18653/v1/2022.emnlp-main.534 ↗
9. Coot: Cooperative hierarchical transformer for video-text representation learning
10. Coordinated-joint translation fusion framework with sentiment-interactive graph convolutional networks for multimodal sentiment analysis10.1016/j.ipm.2023.103538 ↗
11. Co-space Representation Interaction Network for multimodal sentiment analysis10.1016/j.knosys.2023.111149 ↗
12. Truong, Q.T., and Lauw, H.W. (February, January 27). Vistanet: Visual aspect attention network for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
13. An Autoencoder-based Self-Supervised Learning for Multimodal Sentiment Analysis10.1016/j.ins.2024.120682 ↗
14. Integrating multimodal information in large pretrained transformers10.18653/v1/2020.acl-main.214 ↗
15. Liang, Y., Tohti, T., and Hamdulla, A. (2022). Multimodal false information detection method based on Text-CNN and SE module. PLoS ONE, 17.10.1371/journal.pone.0277463 ↗
16. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis10.1109/TAFFC.2022.3172360 ↗
17. Sun, Z., Sarma, P., Sethares, W., and Liang, Y. (2020, January 7–12). Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6431 ↗
18. Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. (2020, January 5–10). Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.343 ↗
19. FDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction10.1016/j.knosys.2024.111965 ↗
20. Joint Objective and Subjective Fuzziness Denoising for Multimodal Sentiment Analysis10.1109/TFUZZ.2024.3405541 ↗
21. Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks10.1016/j.asoc.2023.110494 ↗
22. Liu, S., Liang, Y., and Gitter, A. (February, January 27). Loss-balanced task weighting to reduce negative transfer in multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
23. Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., and Morency, L.P. (2018, January 15–20). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.
24. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages10.1109/MIS.2016.94 ↗
25. Wang, Y., Shen, Y., Liu, Z., Liang, P.P., Zadeh, A., and Morency, L.P. (February, January 27). Words can shift: Dynamically adjusting word representations using nonverbal behaviors. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
26. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv.10.18653/v1/D17-1115 ↗
27. On the comparison between multi-modal and single-modal contrastive learning
28. A comprehensive survey on contrastive learning10.1016/j.neucom.2024.128645 ↗
29. Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., and Liu, J. (2021). Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling. arXiv.10.1109/CVPR46437.2021.00725 ↗
30. Li, L., Chen, Y.C., Cheng, Y., Gan, Z., Yu, L., and Liu, J. (2020). HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training. arXiv.10.18653/v1/2020.emnlp-main.161 ↗
31. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
32. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text
33. Multi-Adaptive Optimization for multi-task learning with deep neural networks10.1016/j.neunet.2023.11.038 ↗
34. Ren, Y., Li, Y., and Kong, A.W.K. (October, January 29). Adaptive Multi-task Learning for Few-Shot Object Detection. Proceedings of the European Conference on Computer Vision, Milan, Italy.
35. Agiza, A., Neseem, M., and Reda, S. (2024, January 16–22). Mtlora: Low-rank adaptation approach for efficient multi-task learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01533 ↗
36. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv.10.18653/v1/P18-1209 ↗
37. Multimodal transformer for unaligned multimodal language sequences10.18653/v1/P19-1656 ↗
38. Hazarika, D., Zimmermann, R., and Poria, S. (2020, January 12–16). Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.
39. Yu, W., Xu, H., Yuan, Z., and Wu, J. (2021, January 2–9). Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i12.17289 ↗
40. Multimodal machine learning: A survey and taxonomy