Image–Text Matching Model Based on CLIP Bimodal Encoding
Source abstract
Abstract
Image–text matching is a fundamental task in the multimodal research field, connecting computer vision and natural language processing by aligning visual content with corresponding textual descriptions. Accurate matching is critical for applications such as image captioning and text-based image retrieval yet remains challenging due to the differences in data modalities. This paper addresses these challenges by proposing a robust image–text matching model inspired by Contrastive Language–Image Pre-training (CLIP). Our approach employs the Vision Transformer (ViT) model as the image encoder and Bidirectional Encoder Representations from Transformers (Bert) as the text encoder, integrating these into a shared vector space to measure semantic similarity. We enhance the model’s training efficiency using the LiT-tuning paradigm to optimize learning through a cosine decay strategy for dynamic adjustment of the learning rate. We validate our method on two benchmark datasets, WuKong and Flickr30k, demonstrating that our model achieves superior performance and significantly improves key evaluation metrics. The results underscore the model’s effectiveness in achieving accurate and robust image–text alignment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130000, China
Funding
Jilin Provincial Department of Education
References (24)
Show source-provided references
1. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–23). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
2. Toyama, J., Misono, M., Suzuki, M., Nakayama, K., and Matsuo, Y. (2016). Neural machine translation with latent semantic of image and text. arXiv.
3. In pursuit of beauty: Aesthetic-aware and context-adaptive photo selection in crowdsensing10.1109/TKDE.2023.3237969 ↗
4. Light-guided and cross-fusion U-Net for anti-illumination image super-resolution10.1109/TCSVT.2022.3194169 ↗
5. Faghri, F., Fleet, D., Kiros, J., and Fidler, S.V. (2017). Improving visual-semantic embeddings with hard negatives. arXiv.
6. Lee, K.-H., Chen, X., Hua, G., Hu, H., and He, X. (2018, January 8–14). Stacked cross attention for image-text matching. Proceedings of the European Conference on Computer Vision (ECCV), 15th European Conference, Munich, Germany.10.1007/978-3-030-01225-0_13 ↗
7. Zhang, Q., Lei, Z., Zhang, Z., and Li, S.Z. (2020, January 13–19). Context-aware attention network for image-text retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00359 ↗
8. Hu, Z., Luo, Y., Lin, J., Yan, Y., and Chen, J. (2019, January 10–16). Multi-level visual-semantic alignments with relation-wise dual attention network for image and text matching. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), Macao, China.10.24963/ijcai.2019/111 ↗
9. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
10. Teney, D., Liu, L., and van Den Hengel, A. (2017, January 21–26). Graph-structured representations for visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.344 ↗
11. Kiros, R., Salakhutdinov, R., and Zemel, R.S. (2014). Unifying visual-semantic embeddings with multimodal neural language models. arXiv.
12. Wei, X., Zhang, T., Li, Y., Zhang, Y., and Wu, F. (2020, January 13–19). Multi-modality cross attention network for image and sentence matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01095 ↗
13. Nam, H., Ha, J.-W., and Kim, J. (2017, January 21–26). Dual attention networks for multimodal reasoning and matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.232 ↗
14. Messina, N., Stefanini, M., Cornia, M., Baraldi, L., Falchi, F., Amato, G., and Cucchiara, R. (2022, January 14–16). Aladin: Distilling fine-grained alignment scores for efficient image-text matching and retrieval. Proceedings of the 19th International Conference on Content-Based Multimedia Indexing, Graz, Austria.10.1145/3549555.3549576 ↗
15. Fu, Z., Mao, Z., Song, Y., and Zhang, Y. (2023, January 17–24). Learning semantic relationship among instances for image-text matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01455 ↗
16. Pan, Z., Wu, F., and Zhang, B. (2023, January 17–24). Fine-grained image-text matching by cross-modal hard aligning network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01847 ↗
17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. arXiv.
18. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
19. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark
20. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., and Lazebnik, S. (2015, January 7–13). Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.303 ↗
21. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
22. Wang, S., Wang, R., Yao, Z., Shan, S., and Chen, X. (2020, January 1–5). Cross-modal scene graph matching for relationship-aware image-text retrieval. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass, CO, USA.10.1109/WACV45572.2020.9093614 ↗
23. Zhang, K., Mao, Z., Wang, Q., and Zhang, Y. (2022, January 18–24). Negative-aware attention framework for image-text matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01521 ↗
24. Wang, H., Zhang, Y., Ji, Z., Pang, Y., and Ma, L. (2020, January 23–28). Consensus-aware visual-semantic embedding for image-text matching. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXIV 16, 2020.