A Review on Vision-Language-Based Approaches: Challenges and Applications
Source abstract
Abstract
In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural language pro... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Media, Religion, Digital Communication · Religious Tourism and Spaces
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Foundations & trends in multimodal machine learning: principles, challenges, and open questions10.1145/3676164 ↗
2. Multimodal deep learning
3. The language, tone and prosody of emotions: neural substrates and dynamics of spoken-word emotion perception10.3389/fnins.2016.00506 ↗
4. Multimodal machine learning: a survey and taxonomy10.1109/TPAMI.2018.2798607 ↗
5. Li · Proceedings of the 39th international conference on machine learning · 162 · 12888
6. Andrew · Proceedings of the 30th international conference on machine learning · 2013 · 28 · 1247
7. Socher · Advances in neural information processing systems · 2013 · 26
8. Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning
9. Visual genome: connecting language and vision using crowdsourced dense image annotations10.1007/s11263-016-0981-7 ↗
10. Radford · Proceedings of the 38th International Conference on Machine Learning, ICML 2021 · 2021 Jul 18–24 · 139 · 8748
11. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv:201011929. 2021.
12. A simple framework for contrastive learning of visual representations
13. Vaswani · Advances in neural information processing systems · 2017 · 30
14. LXMERT: learning cross-modality encoder representations from transformers
15. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
16. Devlin · Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies · 2019 · (Long and Short Papers) · 4171
17. Li LH, Yatskar M, Yin D, Hsieh CJ, Chang KW. Visualbert: a simple and performant baseline for vision and language. arXiv:190803557. 2019.
18. Chen · UNITER: universal image-text representation learning · 2020 · 104
19. Unified vision-language pre-training for image captioning and VQA
20. Li · Proceedings of the 40th international conference on machine learning · 2023 · 202 · 19730
21. VinVL: revisiting visual representations in vision-language models
22. SimVLM: simple visual language model pretraining with weak supervision
23. Yang C, Zhu X, Zhu J, Su W, Wang J, Dong X, et al. Vision model pre-training on interleaved image-text data via latent compression learning. arXiv:240607543. 2024.
24. RegionCLIP: region-based language-image pretraining
25. Multimodal few-shot learning with frozen language models
26. Bottom-up and top-down attention for image captioning and visual question answering
27. VILA: on pre-training for visual language models
28. Zheng · Computer vision–ECCV 2024 · 2025 · 7310.1007/978-3-031-72649-1_5 ↗
29. Liu · Advances in neural information processing systems · 2023 · 36 · 34892
30. Anchor-based robust finetuning of vision-language models
31. Jin · Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long Papers) · 2022 · 276310.18653/v1/2022.acl-long.197 ↗
32. Guo Y, Zhang H, Wong Y, Nie L, Kankanhalli M. Elip: efficient language-image pre-training with fewer vision tokens. arXiv:230916738. 2023.
33. Bootstrapping vision-language learning with decoupled language pre-training
34. ProtoCLIP: prototypical contrastive language image pretraining
35. CapsFusion: rethinking image-text data at scale
36. Purushwalkam S, Gokul A, Joty S, Naik N. Bootpig: bootstrapping zero-shot personalized image generation capabilities in pretrained diffusion models. arXiv:240113974. 2024.10.1007/978-3-031-91907-7_15 ↗
37. Liang · Centered masking for language-image pre-training · 2024 · 90
38. Filtering, distillation, and hard negatives for vision-language pre-training
39. EmbodiedGPT: vision-language pre-training via embodied chain of thought
40. Scaling language-image pre-training via masking
41. Mohammed · Proceedings of the 14th Asian Conference on Machine Learning · 2023 · 189 · 738
42. Xu · A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model · 2022 · 736
43. NLIP: noise-robust language-image pre-training
44. Chen X, Fang H, Lin T, Vedantam R, Gupta S, Dollár P, et al. Microsoft COCO captions: data collection and evaluation server. arXiv:1504.00325. 2015.
45. Bharne · RAiSE-2023 · 2024 · 51 · 18210.3390/engproc2023059182 ↗
46. Bootstrapping interactive image-text alignment for remote sensing image captioning
47. MedKLIP: medical knowledge enhanced language-image pre-training for X-ray diagnosis
48. Self-chained image-language model for video localization and question answering
49. He S, Nie Y, Chen Z, Cai Z, Wang H, Yang S, et al. MedDr: diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv:240415127. 2024.
50. VQA: visual question answering
51. Su W, Zhu X, Cao Y, Li B, Lu L, Wei F, et al. Vl-bert: pre-training of generic visual-linguistic representations. arXiv:190808530. 2019.
52. Language models with image descriptors are strong few-shot video-language learners
53. Distilling vision-language models on millions of videos
54. SpectralGPT: spectral remote sensing foundation model10.1109/TPAMI.2024.3362475 ↗
55. UIU-Net: U-Net in U-Net for infrared small object detection10.1109/TIP.2022.3228497 ↗
56. LRRNet: a novel representation learning guided fusion network for infrared and visible images10.1109/TPAMI.2023.3268209 ↗
57. Duan J, Yuan W, Pumacay W, Wang YR, Ehsani K, Fox D, et al. Manipulate-anything: automating real-world robots using vision-language models. arXiv:240618915. 2024.
58. Guo Z, Lykov A, Yagudin Z, Konenkov M, Tsetserukou D. Co-driver: VLM-based autonomous driving assistant with human-like behavior and understanding for complex road scenes. arXiv:240505885. 2024.10.1109/FLLM63129.2024.10852498 ↗
59. Hao X, Chen W, Yan Y, Zhong S, Wang K, Wen Q, et al. UrbanVLP: a multi-granularity vision-language pre-trained foundation model for urban indicator prediction. arXiv:240316831. 2024.10.1609/aaai.v39i27.35024 ↗
60. Gwinnup J, Duh K. A survey of vision-language pre-training from the lens of multimodal machine translation. arXiv:230607198. 2023.
61. Lin · Computer vision–ECCV 2014 · 2014 · 74010.1007/978-3-319-10602-1_48 ↗
62. Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models
63. Making the V in VQA matter: elevating the role of image understanding in visual question answering
64. BLEU: a method for automatic evaluation of machine translation
65. CIDEr: consensus-based image description evaluation
66. Banerjee · Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization · 2005 · 65
67. Anderson · Computer vision–ECCV 2016 · 2016 · 38210.1007/978-3-319-46454-1_24 ↗
68. Nocaps: novel object captioning at scale
69. Coarse-to-fine vision-language pre-training with fusion in the backbone
70. GQA: a new dataset for real-world visual reasoning and compositional question answering
71. Vision-language pre-training with triple contrastive learning
72. Object prior embedded network for query-agnostic image retrieval
73. ImageNet: a large-scale hierarchical image database
74. An empirical study of GPT-3 for few-shot knowledge-based VQA
75. OK-VQA: a visual question answering benchmark requiring external knowledge
76. Hou Z, Sun F, Chen YK, Xie Y, Kung SY. MILAN: masked image pretraining on language assisted representation. arXiv:220806049. 2022.
77. You · Learning visual representation from modality-shared contrastive language-image pre-training · 2022 · 69
78. GrowCLIP: data-aware automatic model growing for large-scale contrastive language-image pre-training
79. LVIS: a dataset for large vocabulary instance segmentation
80. Flamingo: a visual language model for few-shot learning
81. Lu · Advances in neural information processing systems · 2022 · 35 · 2507
82. VizWiz-Priv: a dataset for recognizing the presence and purpose of private visual information in images taken by blind people
83. Towards VQA models that can read
84. The dark side of dataset scaling: evaluating racial classification in multimodal models
85. Chen · Computer vision–ACCV 2024 · 2025 · 9810.1007/978-981-96-0908-6_6 ↗
86. Scaling up visual and vision-language representation learning with noisy text supervision
87. Chen · Findings of the association for computational linguistics: ACL 2023 · 2023 · 8666
88. Ko B, Gu G. Large-scale bilingual language-image contrastive learning. arXiv:220314463. 2022.
89. Wu B, Cheng R, Zhang P, Gao T, Vajda P, Gonzalez JE. Data efficient language-supervised zero-shot recognition with optimal transport distillation. arXiv:211209445. 2021.10.1109/CVPRW53098.2021.00348 ↗
90. Li Y, Liang F, Zhao L, Cui Y, Ouyang W, Shao J, et al. Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. arXiv:211005208. 2021.
91. Improving CLIP training with language rewrites
92. ALIP: adaptive language-image pre-training with synthetic caption
93. Hu X, Liu D, Li H, Huang X, Shao J. VLSBench: unveiling visual leakage in multimodal safety. arXiv:241119939. 2024.10.18653/v1/2025.acl-long.405 ↗
94. Zhang Y, Chen L, Zheng G, Gao Y, Zheng R, Fu J, et al. SPA-VL: a comprehensive safety preference alignment dataset for vision language model. arXiv:240612030. 2024.10.1109/CVPR52734.2025.01850 ↗
95. Li L, Xie Z, Li M, Chen S, Wang P, Chen L, et al. VLFeedback: a large-scale AI feedback dataset for large vision-language models alignment. arXiv:241009421. 2024.10.18653/v1/2024.emnlp-main.358 ↗
96. Conditional prompt learning for vision-language models
97. Learning to prompt for open-vocabulary object detection with vision-language model
98. Huang H, Chu J, Wei F. Unsupervised prompt learning for vision-language models. 2022. doi: 10.48550/arXiv.2204.03649.
99. Brown · Advances in neural information processing systems · 2020 · 33 · 1877
100. On the dangers of stochastic parrots: can language models be too big?