Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification
Source abstract
Abstract
Multimodal Sentiment Analysis (SA) is gaining popularity due to its broad application potential. The existing studies have focused on the SA of single modalities, such as texts or photos, posing challenges in effectively handling social media data with multiple modalities. Moreover, most multimodal research has concentrated on merely combining the two modalities rather than exploring their complex correlations, leading to unsatisfactory sentiment classification results. Motivated by this, we propose a new visual-textual sentiment classification model named Multi-Model Fusion (MMF), which uses a mixed fusion framework for SA to effectively capture the essential information and the intrinsic relationship between the visual and textual content. The proposed model comprises three deep neural networks. Two different neural networks are proposed to extract the most emotionally relevant aspects of image and text data. Thus, more discriminative features are gathered for accurate sentiment classification. Then, a multichannel joint fusion model with a self-attention technique is proposed to exploit the intrinsic correlation between visual and textual characteristics and obtain emotionally rich information for joint sentiment classification. Finally, the results of the three classifiers are integrated using a decision fusion scheme to improve the robustness and generalizability of the proposed model. An interpretable visual-textual sentiment classification model is further developed using the Local Interpretable Model-agnostic Explanation model (LIME) to ensure the model’s explainability and resilience. The proposed MMF model has been tested on four real-world sentiment datasets, achieving (99.78%) accuracy on Binary_Getty (BG), (99.12%) on Binary_iStock (BIS), (95.70%) on Twitter, and (79.06%) on the Multi-View Sentiment Analysis (MVSA) dataset. These results demonstrate the superior performance of our MMF model compared to single-model approaches and current state-of-the-art techniques based on model evaluation criteria.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Visual Attention and Saliency Detection · Sentiment Analysis and Opinion Mining · Advanced Computing and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (75)
Show source-provided references
1. A hybrid model integrating deep learning with investor sentiment analysis for stock price prediction10.1016/j.eswa.2021.115019 ↗
2. A deep learning-based approach to constructing a domain sentiment lexicon: A case study in financial distress prediction10.1016/j.ipm.2021.102673 ↗
3. Sentiment analysis of political communication: Combining a dictionary approach with crowdcoding
4. Sentiment analysis of Covid-19 tweets by deep learning classifiers—a study to show how popularity is affecting accuracy in social media
5. Tourism recommendation system based on semantic clustering and sentiment analysis10.1016/j.eswa.2020.114324 ↗
6. Distributed representations of sentences and documents
7. Sentiment analysis of customers’ reviews using a hybrid evolutionary svm-based approach in an imbalanced data distribution10.1109/ACCESS.2022.3149482 ↗
8. Transformer-based deep learning models for the sentiment analysis of social media data10.1016/j.array.2022.100157 ↗
9. ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis10.1016/j.future.2020.08.005 ↗
10. Sigmoidal particle swarm optimization for twitter sentiment analysis10.32604/cmc.2023.031867 ↗
11. Optimized support vector machine model for visual sentiment analysis
12. Automatic visual sentiment analysis with convolution neural network
13. Visual sentiment analysis with active learning10.1109/ACCESS.2020.3024948 ↗
14. Multi-level context pyramid network for visual sentiment analysis10.3390/s21062136 ↗
15. A deep learning architecture of RA-DLNet for visual sentiment analysis10.1007/s00530-020-00656-7 ↗
16. Region-based convolutional neural network using group sparse regularization for image sentiment classification10.1186/s13640-019-0433-8 ↗
17. I. K. Al-Tameemi, M. R. F. Derakhshi, S. Pashazadeh and M. Asadpour, “A comprehensive review of visual-textual sentiment analysis from social media networks,” arXiv preprint arXiv: 2207. 02160, 2022.
18. A novel visual-textual sentiment analysis framework for social media data10.1007/s12559-021-09929-3 ↗
19. Cross-modal image sentiment analysis via deep correlation of textual semantic10.1016/j.knosys.2021.106803 ↗
20. Attention-based modality-gated networks for image-text
21. Visual-textual sentiment classification with bi-directional multi-level attention networks10.1016/j.knosys.2019.04.018 ↗
22. Various syncretic co-attention network for multimodal sentiment analysis10.1002/cpe.5954 ↗
23. Global local fusion neural network for multimodal sentiment analysis
24. Improving targeted multimodal sentiment classification with semantic description of images10.32604/cmc.2023.038220 ↗
25. Multi-modal fusion network with complementarity and importance for emotion recognition10.1016/j.ins.2022.11.076 ↗
26. VABDC-Net: A framework for visual-caption sentiment recognition via spatio-depth visual attention and bi-directional caption processing10.1016/j.knosys.2023.110515 ↗
27. Visual-textual sentiment analysis enhanced by hierarchical cross-modality interaction10.1109/JSYST.2020.3026879 ↗
28. A deep multi-level attentive network for multimodal sentiment analysis10.1145/3517139 ↗
29. Ensemble transfer learning-based multimodal sentiment analysis using weighted convolutional neural networks10.1016/j.ipm.2022.102929 ↗
30. Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data10.1016/j.ipm.2019.102141 ↗
31. Sentiment analysis of multimodal twitter data10.1007/s11042-019-7390-1 ↗
32. Psychological foundations of explainability and interpretability in artificial intelligence
33. Towards the explainability of multimodal speech emotion recognition
34. A unified approach to interpreting model predictions
35. Why should I trust you?’ explaining the predictions of any classifier
36. Beyond human: Deep learning, explainability and representation10.1177/0263276420966386 ↗
37. Learning important features through propagating activation differences
38. Multimodal language analysis in the wild: CMU-mosei dataset and interpretable dynamic fusion graph
39. Do explanations make VQA models more predictable to a human?
40. Interpretable multimodal emotion recognition using hybrid fusion of speech and image data
41. P. Kumar, S. Malik and B. Raman, “Hybrid fusion based interpretable multimodal emotion recognition with insufficient labelled data,” arXiv: 2208.11450, 2022.
42. Employing co-learning to evaluate the explainability of multimodal sentiment analysis
43. DIME: Fine-grained interpretations of multimodal models via disentangled local explanations
44. S. Han, “Googletrans · PyPI,” Available: https://pypi.org/project/googletrans/
45. Efficient estimation of word representations in vector space
46. GloVe: Global vectors for word representation
47. BERT: Pre-training of deep bidirectional transformers for language understanding
48. Attention is all you need
49. On the properties of neural machine translation: Encoder-decoder approaches
50. J. Chung, C. Gulcehre, K. Cho and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv: 1412. 3555, 2014.
51. Rethinking the inception architecture for computer vision
52. Going deeper with convolutions
53. Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation
54. Attnsense: Multi-level attention mechanism for multimodal human activity recognition
55. SLIC superpixels compared to state-of-the-art superpixel methods
56. “Royalty free stock photos, illustrations, vector art, and video clips—getty images,” [Online] Available: https://www.gettyimages.com/
57. Large-scale visual sentiment ontology and detectors using adjective noun pairs
58. VADER: A parsimonious rule-based model for sentiment analysis of social media text
59. C. J. Hutto and E. Gilbert, “GitHub—cjhutto/vaderSentiment: VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains,” [Online] Available: https://github.com/cjhutto/vaderSentiment
60. “Stock images, royalty-free pictures, illustrations & videos—iStock,” [Online] Available: https://www.istockphoto.com/
61. “Use cases, tutorials, & documentation | Twitter developer platform,” [Online] Available: https://developer.twitter.com/en
62. L. Vadicamo, F. Carrara, A. Cimino, S. Cresci, F. Dell'Orletta et al., “GitHub—fabiocarrara/visual-sentiment-analysis: PyTorch port of models for Visual Sentiment Analysis pre-trained on the T4SA dataset,” [Online] Available: https://github.com/fabiocarrara/visual-sentiment-analysis
63. Cross-media learning for image sentiment analysis in the wild
64. Sentiment analysis on multi-view social data
65. MultiSentiNet: A deep semantic network for multimodal sentiment analysis
66. Convolutional neural networks for sentence classification
67. Very deep convolutional networks for large-scale image recognition
68. Deep residual learning for image recognition
69. Adapting BERT for target-oriented multimodal sentiment classification
70. Cross-modal sentiment sensing with visual-augmented representation and diverse decision fusion
71. Exploiting BERT for multimodal target sentiment classification through input space translation
72. A co-memory network for multimodal sentiment analysis
73. Cross-modal complementary network with hierarchical fusion for multimodal sentiment classification10.26599/TST.2021.9010055 ↗
74. Multimodal sentiment analysis with image-text interaction network
75. Z. Li, B. Xu, C. Zhu and T. Zhao, “CLMLF: A contrastive learning and multi-layer fusion method for multimodal sentiment detection,” arXiv: 2204. 05515, 2022.10.18653/v1/2022.findings-naacl.175 ↗