A global–local dual-stream collaborative enhancement model for cross-modal image and text retrieval in remote sensing scenarios
Source abstract
Abstract
Remote sensing image-text cross-modal retrieval technology serves as a key supporting technology for transforming remote sensing big data into knowledge services, and it holds significant application value in national strategic areas such as smart city development and ecological environment protection. To address the limitations of single feature representation caused by the multimodal object types and complex texture spatial structures of remote sensing images, this paper proposes a cross-modal retrieval framework based on global–local feature collaborative enhancement. In terms of visual representation, a local feature branch using graph convolutional networks is introduced based on contrastive language-image pre-training global features, along with a local–global feature synergistic enhancement module designed to achieve self-attention enhancement of multi-scale features, complementary calibration, and hierarchical fusion through an adaptive weight allocation strategy. For text representation, an attention-convolution fusion module is designed to enhance the semantics of the text and align it with visual information. Finally, cosine similarity is employed to compute the semantic correlation between visual and text features, and contrastive loss is used for model optimization. Experiments on the RSITMD and RSICD datasets verify the effectiveness of the proposed method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Engineering, Hebei University of Architecture 1 , Zhangjiakou 075000,
Hebei MinZu Normal University & The Technology Innovation Center of Cultural Tourism Big Data of Hebei Province 2 , Chengde 067000,
Shalingzi Hospital 3 , Zhangjiakou 075000,
Funding
Jiaqi Ji · Jiuyi Zhang
References (40)
Show source-provided references
1. An empirical study of remote sensing pretraining10.1109/TGRS.2022.3176603 ↗
2. Comparative analysis on cross-modal information retrieval: A review10.1016/j.cosrev.2020.100336 ↗
3. Image retrieval from remote sensing big data: A survey10.1016/j.inffus.2020.10.008 ↗
4. Unsupervised remote sensing image retrieval using probabilistic latent semantic hashing10.1109/LGRS.2020.2969491 ↗
5. Deep cross-modal image–voice retrieval in remote sensing10.1109/tgrs.2020.2979273 ↗
6. Deep semantic understanding of high resolution remote sensing image
7. Exploring models and data for remote sensing image caption generation10.1109/TGRS.2017.2776321 ↗
8. Z. Yuan , W.Zhang, K.Fu, X.Li, C.Deng, H.Wang, and X.Sun, “Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval,” arXiv:2204.09868 (2022).10.1109/TGRS.2021.3078451 ↗
9. Multiscale multiinteraction network for remote sensing image captioning10.1109/JSTARS.2022.3153636 ↗
10. Remote-sensing image captioning based on multilayer aggregated transformer10.1109/LGRS.2022.3150957 ↗
11. A transformer-based cross-modal image-text retrieval method using feature decoupling and reconstruction
12. Knowledge-aware text-image retrieval for remote sensing images10.1109/TGRS.2024.3486977 ↗
13. MCRN: A multi-source cross-modal retrieval network for remote sensing10.1016/j.jag.2022.103071 ↗
14. Deep semantic understanding of high resolution remote sensing image
15. Can a machine generate humanlike language descriptions for a remote sensing image?10.1109/tgrs.2017.2677464 ↗
16. Exploring models and data for remote sensing image caption generation10.1109/tgrs.2017.2776321 ↗
17. SD-RSIC: Summarization-driven deep remote sensing image captioning10.1109/TGRS.2020.3031111 ↗
18. Multi-label semantic feature fusion for remote sensing image captioning10.1016/j.isprsjprs.2021.11.020 ↗
19. A novel SVM-based decoder for remote sensing image captioning10.1109/TGRS.2021.3105004 ↗
20. Fusion-based correlation learning model for cross-modal remote sensing image retrieval10.1109/lgrs.2021.3131592 ↗
21. TextRS: Deep bidirectional triplet network for matching text to remote sensing images10.3390/rs12030405 ↗
22. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval10.1109/TGRS.2021.3078451 ↗
23. Ground-based remote sensing cloud classification via context graph attention network10.1109/tgrs.2021.3063255 ↗
24. Text-image matching for cross-modal remote sensing image retrieval via graph neural network10.1109/jstars.2022.3231851 ↗
25. Knowledge-aided momentum contrastive learning for remote-sensing image text retrieval10.1109/TGRS.2023.3332317 ↗
26. A lightweight multi-scale crossmodal text-image retrieval method in remote sensing10.1109/tgrs.2021.3124252 ↗
27. Learning transferable visual models from natural language supervision
28. F. Liu , D.Chen, Z.Guan, X.Zhou, J.Zhu, and J.Zhou, “RemoteCLIP: A vision language foundation model for remote sensing,” arXiv:2306.11029 (2023).10.1109/TGRS.2024.3390838 ↗
29. Z. Wang , R.Prabha, T.Huang, J.Wu, and R.Rajagopal, “SkyScript: A large and semantically diverse vision-language dataset for remote sensing,” arXiv:2312.12856 (2023).10.1609/aaai.v38i6.28393 ↗
30. Rs5m: A large scale vision-language dataset for remote sensing vision-language foundation model
31. L. Djoufack Basso , “CLIP-RS: A cross-modal remote sensing image retrieval based on clip, anorthern virginia case study,” Ph.D. dissertation (Virginia Tech, 2022).
32. A multiscale grouping transformer with CLIP latents for remote sensing image captioning10.1109/tgrs.2024.3385500 ↗
33. Learning semantic concepts and order for image and sentence matching
34. F. Faghri , D. J.Fleet, J. R.Kiros, and S.Fidler, “Vse++: Improving visual-semantic embeddings with hard negatives,” arXiv:1707.05612 (2017).
35. Stacked cross attention for image-text matching
36. Camp: Cross-modal adaptive message passing for text-image retrieval
37. Remote sensing cross-modal text-image retrieval based on global and local information10.1109/tgrs.2022.3163706 ↗
38. Exploring uni-modal feature learning on entities and relations for remote sensing cross-modal text-image retrieval10.1109/tgrs.2023.3333375 ↗
39. Hypersphere-based remote sensing cross-modal text–image retrieval via curriculum learning10.1109/tgrs.2023.3318227 ↗
40. Learning transferable visual models from natural language supervision