Globally Guided Confidence Enhancement Network for Image-Text Matching
Source abstract
Abstract
Image-text matching is a crucial aspect of multi-modal intelligence. The main challenge in this area is accurately measuring the relevance between the image and text, using evidence obtained through matching. Previous studies either concentrated on obtaining a well-represented global feature to measure similarity directly or on investigating complex matching patterns at a local level before aggregating them, with little attention paid to combining them. We propose a Globally Guided Confidence Enhancement Network that combines both approaches by obtaining a good global representation to guide fine-grained local interactions. In this process, content that better matches the text from a global perspective is enhanced and represented with confidence scores. Extensive experiments demonstrate that the approach we have employed achieves superior performance on Flickr30K and MSCOCO datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, College of Information Science and Engineering, Xinjiang University, Urumqi 830017, China
Funding
Natural Science Foundation of Autonomous · Autonomous Region High-Level Innovative Talent Project
References (37)
Show source-provided references
1. Toyama, J., Misono, M., Suzuki, M., Nakayama, K., and Matsuo, Y. (2023, February 14). Neural Machine Translation with Latent Semantic of Image and Text. Available online: https://arxiv.53yu.com/abs/1611.08459v1.
2. Nam, H., Ha, J.-W., and Kim, J. (2017, January 21–26). Dual Attention Networks for Multimodal Reasoning and Matching. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.232 ↗
3. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–22). AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
4. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
5. Gu, J., Cai, J., Joty, S., Niu, L., and Wang, G. (2018, January 18–22). Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00750 ↗
6. Huang, Y., Wang, W., and Wang, L. (2017, January 21–26). Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.767 ↗
7. Liu, Y., Guo, Y., Bakker, E.M., and Lew, M.S. (2017, January 22–29). Learning a Recurrent Residual Fusion Network for Multimodal Matching. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.442 ↗
8. Shi, B., Ji, L., Lu, P., Niu, Z., and Duan, N. (2019, January 10–16). Knowledge Aware Semantic Concept Expansion for Image-Text Matching. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Macao, China.10.24963/ijcai.2019/720 ↗
9. Wang, L., Li, Y., and Lazebnik, S. (2016, January 27–30). Learning Deep Structure-Preserving Image-Text Embeddings. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.541 ↗
10. Li, K., Zhang, Y., Li, K., Li, Y., and Fu, Y. (November, January 27). Visual Semantic Reasoning for Image-Text Matching. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV 2019), Seoul, Republic of Korea.10.1109/ICCV.2019.00475 ↗
11. Deep Collaborative Multi-View Hashing for Large-Scale Image Search10.1109/TIP.2020.2974065 ↗
12. Stacked Cross Attention for Image-Text Matching10.1007/978-3-030-01225-0_13 ↗
13. Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., and Han, J. (2020, January 13–19). IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01267 ↗
14. Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching
15. Hu, Z., Luo, Y., Lin, J., Yan, Y., and Chen, J. (2019, January 10–16). Multi-Level Visual-Semantic Alignments with Relation-Wise Dual Attention Network for Image and Text Matching. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Macao, China.10.24963/ijcai.2019/111 ↗
16. Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., and Shao, J. (November, January 27). CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00586 ↗
17. Adaptive Cross-Modal Embeddings for Image-Text Alignment
18. Wu, Y., Wang, S., Song, G., and Huang, Q. (2019, January 21–25). Learning Fragment Self-Attention Embeddings for Image-Text Matching. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350940 ↗
19. Multimodal Intelligence: Representation Learning, Information Fusion, and Applications10.1109/JSTSP.2020.2987728 ↗
20. Similarity Reasoning and Filtration for Image-Text Matching
21. Liu, C., Mao, Z., Zhang, T., Xie, H., Wang, B., and Zhang, Y. (2020). Graph Structured Network for Image-Text Matching. arXiv.10.1109/CVPR42600.2020.01093 ↗
22. Show Your Faith: Cross-Modal Confidence-Aware Network for Image-Text Matching
23. Chen, J., Hu, H., Wu, H., Jiang, Y., and Wang, C. (2021, January 20–25). Learning the Best Pooling Strategy for Visual Semantic Embedding. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01553 ↗
24. Scalable Deep Hashing for Large-Scale Social Image Retrieval10.1109/TIP.2019.2940693 ↗
25. Faghri, F., Fleet, D.J., Kiros, J.R., and Fidler, S. (2018). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. arXiv.
26. Analysis on Image Enhancement Techniques10.5815/ijem.2023.02.02 ↗
27. Machine Learning in Cyberbullying Detection from Social-Media Image or Screenshot with Optical Character Recognition
28. Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J., Li, B., and Fan, X. (2019, January 10–16). Position Focused Attention Network for Image-Text Matching. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Macao, China.10.24963/ijcai.2019/526 ↗
29. Kipf, T.N., and Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. arXiv.
30. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations10.1007/s11263-016-0981-7 ↗
33. Bidirectional Recurrent Neural Networks10.1109/78.650093 ↗
34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Advances in Neural Information Processing Systems, Curran Associates, Inc.
35. From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions10.1162/tacl_a_00166 ↗
36. Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., and Dollár, P. (2015). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
37. Kingma, D.P., and Ba, J. (2017). Adam: A Method for Stochastic Optimization. arXiv.