journal article·2025Open access

GLAMSNet: A Gated-Linear Aspect-Aware Multimodal Sentiment Network with Alignment Supervision and External Knowledge Guidance

Dan Wang, Zhoubin Li, Yuze Xia, Zhenhua Yu

Computers, Materials & Continua· Volume 85· Issue 3

Abstract

Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to detect sentiment polarity toward specific aspects by leveraging both textual and visual inputs. However, existing models suffer from weak aspect-image alignment, modality imbalance dominated by textual signals, and limited reasoning for implicit or ambiguous sentiments requiring external knowledge. To address these issues, we propose a unified framework named Gated-Linear Aspect-Aware Multimodal Sentiment Network (GLAMSNet). First of all, an input encoding module is employed to construct modality-specific and aspect-aware representations. Subsequently, we introduce an image–aspect correlation matching module to provide hierarchical supervision for visual-textual alignment. Building upon these components, we further design a Gated-Linear Aspect-Aware Fusion (GLAF) module to enhance aspect-aware representation learning by adaptively filtering irrelevant textual information and refining semantic alignment under aspect guidance. Additionally, an External Language Model Knowledge-Guided mechanism is integrated to incorporate sentiment-aware prior knowledge from GPT-4o, enabling robust semantic reasoning especially under noisy or ambiguous inputs. Experimental studies conducted based on Twitter-15 and Twitter-17 datasets demonstrate that the proposed model outperforms most state-of-the-art methods, achieving 79.36% accuracy and 74.72% F1-score, and 74.31% accuracy and 72.01% F1-score, respectively.
Source: openalex_inverted_index · Quality: baseline

Research context

OpenAlex topics: Sentiment Analysis and Opinion Mining · Text and Document Classification Technologies · Multimodal Machine Learning Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

References (45)

Show source-provided references
1. Issues and challenges of aspect-based sentiment analysis: a comprehensive survey10.1109/TAFFC.2020.2970399 ↗
2. Attention-based LSTM for aspect-level sentiment classification
3. Interactive attention networks for aspect-level sentiment classification
4. Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018. doi:10.48550/arXiv.1810.04805.
5. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461. 2019. doi:10.48550/arXiv.1910.13461.10.18653/v1/2020.acl-main.703 ↗
6. Low-rank adapter layers and bidirectional gated feature fusion for multimodal hateful memes classification
7. Fake news detection based on cross-modal ambiguity computation and multi-scale feature fusion
8. Multimodal aspect-based sentiment analysis: a survey of tasks, methods, challenges and future directions10.1016/j.inffus.2024.102552 ↗
9. Exploring the effectiveness of machine learning and deep learning algorithms for sentiment analysis: a systematic literature review
10. X-OODM: leveraging explainable object-oriented design methodology for multi-domain sentiment analysis
11. Multi-interactive memory network for aspect based multimodal sentiment analysis
12. Adapting BERT for target-oriented multimodal sentiment classification
13. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification10.1109/TASLP.2019.2957872 ↗
14. Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis10.1109/TAFFC.2022.3171091 ↗
15. Learning from adjective-noun pairs: a knowledge-enhanced framework for target-oriented multimodal sentiment classification
16. Aspect-level multimodal sentiment analysis based on co-attention fusion10.1007/s41060-023-00497-3 ↗
17. Recurrent attention network on memory for aspect sentiment analysis
18. Transformation networks for target-oriented sentiment classification
19. Multi-grained attention network for aspect-level sentiment classification
20. Deep residual learning for image recognition
21. Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis10.1007/s10462-023-10685-z ↗
22. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
23. Exploiting BERT for multimodal target sentiment classification through input space translation
24. SaliencyBERT: recurrent attention network for target-oriented multimodal sentiment classification
25. Face-sensitive image-to-emotional-text cross-modal translation for multimodal aspect-based sentiment analysis
26. Affective region recognition and fusion network for target-level multimodal sentiment classification10.1109/TETC.2022.3231746 ↗
27. Dual causes generation assisted model for multimodal aspect-based sentiment classification10.1109/TNNLS.2024.3415028 ↗
28. A novel energy based model mechanism for multi-modal aspect-based sentiment analysis
29. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis10.1109/TAFFC.2023.3274829 ↗
30. Contrastive learning based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities10.1109/TAFFC.2024.3378570 ↗
31. Exploit the visual sentiment of the item images to fuse with textual sentiment in context aware collaborative filtering10.1016/j.eswa.2024.125970 ↗
32. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692. 2019. doi:10.48550/arXiv.1907.11692.
33. Faster R-CNN: towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
34. He P, Liu X, Gao J, Chen W. DeBERTa: decoding-enhanced BERT with disentangled attention. arXiv:2006.03654. 2020. doi:10.48550/arxiv.2006.03654.
35. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv:2010.11929. 2020. doi:10.48550/arXiv.2010.11929.
36. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. arXiv:2103.00020. 2021. doi:10.48550/arxiv.2103.00020.
37. Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation
38. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching
39. Multimodal transformer for unaligned multimodal language sequences
40. Rethinking diversified and discriminative proposal generation for visual grounding
41. TVQA+: spatio-temporal grounding for video question answering
42. EfficientViT: lightweight multi-scale attention for high-resolution dense prediction
43. Visual attention model for name tagging in multimodal social media
44. Adaptive co-attention network for named entity recognition in tweets
45. Attention guided CAM: visual explanations of vision transformer guided by self-attention