Contrasting Dual Transformer Architectures for Multi-Modal Remote Sensing Image Retrieval
Source abstract
Abstract
Remote sensing technology has advanced rapidly in recent years. Because of the deployment of quantitative and qualitative sensors, as well as the evolution of powerful hardware and software platforms, it powers a wide range of civilian and military applications. This in turn leads to the availability of large data volumes suitable for a broad range of applications such as monitoring climate change. Yet, processing, retrieving, and mining large data are challenging. Usually, content-based remote sensing image (RS) retrieval approaches rely on a query image to retrieve relevant images from the dataset. To increase the flexibility of the retrieval experience, cross-modal representations based on text–image pairs are gaining popularity. Indeed, combining text and image domains is regarded as one of the next frontiers in RS image retrieval. Yet, aligning text to the content of RS images is particularly challenging due to the visual-sematic discrepancy between language and vision worlds. In this work, we propose different architectures based on vision and language transformers for text-to-image and image-to-text retrieval. Extensive experimental results on four different datasets, namely TextRS, Merced, Sydney, and RSICD datasets are reported and discussed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Remote-Sensing Image Classification · Image Retrieval and Classification Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia
Center of Smart Robotics Research, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
Department of Computer Science, Community College, King Saud University, Riyadh 11437, Saudi Arabia
Digital Industry Center, Technologies of Vision Unit, Fondazione Bruno Kessler, 38123 Trento, Italy
Funding
King Saud University
References (34)
Show source-provided references
1. Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities10.1109/JSTARS.2020.3005403 ↗
2. Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective10.1109/JSTARS.2020.3013818 ↗
3. Mapping Crop Types in Complex Farming Areas Using SAR Imagery with Dynamic Time Warping10.1016/j.isprsjprs.2021.03.004 ↗
4. A Decision-Level Fusion Approach to Tree Species Classification from Multi-Source Remotely Sensed Data10.1016/j.ophoto.2021.100002 ↗
5. M3C2-EP: Pushing the Limits of 3D Topographic Point Cloud Change Detection by Error Propagation10.1016/j.isprsjprs.2021.06.011 ↗
6. Multi-Attention Fusion and Fine-Grained Alignment for Bidirectional Image-Sentence Retrieval in Remote Sensing10.1109/JAS.2022.105773 ↗
7. NWPU-Captions Dataset and MLCA-Net for Remote Sensing Image Captioning10.1109/TGRS.2022.3201474 ↗
8. Bazi, Y., Bashmal, L., Rahhal, M.M.A., Dayil, R.A., and Ajlan, N.A. (2021). Vision Transformers for Remote Sensing Image Classification. Remote Sens., 13.10.3390/rs13030516 ↗
9. Bashmal, L., Bazi, Y., Al Rahhal, M.M., Alhichri, H., and Al Ajlan, N. (2021). UAV Image Multi-Labeling with Data-Efficient Transformers. Appl. Sci., 11.10.3390/app11093974 ↗
10. Remote Sensing Image Change Detection With Transformers
11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Curran Associates Inc.
12. Language Models Are Few-Shot Learners
13. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models Are Unsupervised Multitask Learners. Technical Report, OpenAI.
14. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv.
15. Hendrycks, D., and Gimpel, K. (2020). Gaussian Error Linear Units (GELUs). arXiv.
16. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Event.
17. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging Properties in Self-Supervised Vision Transformers. arXiv.10.1109/ICCV48922.2021.00951 ↗
18. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
19. Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. arXiv.10.18653/v1/2021.emnlp-main.552 ↗
20. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2020). Training Data-Efficient Image Transformers & Distillation through Attention. arXiv.
21. Abdullah, T., Bazi, Y., Al Rahhal, M.M., Mekhalfi, M.L., Rangarajan, L., and Zuair, M. (2020). TextRS: Deep Bidirectional Triplet Network for Matching Text to Remote Sensing Images. Remote Sens., 12.10.3390/rs12030405 ↗
22. Qu, B., Li, X., Tao, D., and Lu, X. (2016, January 6–8). Deep Semantic Understanding of High Resolution Remote Sensing Image. Proceedings of the 2016 International Conference on Computer, Information and Telecommunication Systems (CITS), Kunming, China.10.1109/CITS.2016.7546397 ↗
23. Exploring Models and Data for Remote Sensing Image Caption Generation10.1109/TGRS.2017.2776321 ↗
24. AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification10.1109/TGRS.2017.2685945 ↗
25. PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval10.1016/j.isprsjprs.2018.01.004 ↗
26. Yang, Y., and Newsam, S. (2010, January 2–5). Bag-of-Visual-Words and Spatial Extensions for Land-Use Classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.10.1145/1869790.1869829 ↗
27. Remote Sensing Image Scene Classification: Benchmark and State of the Art10.1109/JPROC.2017.2675998 ↗
28. Rahhal, M.M.A., Bazi, Y., Abdullah, T., Mekhalfi, M.L., and Zuair, M. (2020). Deep Unsupervised Embedding for Remote Sensing Image Retrieval Using Textual Cues. Appl. Sci., 10.10.3390/app10248931 ↗
29. Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
30. A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing10.1109/JSTARS.2021.3070872 ↗
31. Faghri, F., Fleet, D.J., Kiros, J.R., and Fidler, S. (2017). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. arXiv.
32. Lee, K.-H., Chen, X., Hua, G., Hu, H., and He, X. (2018). Stacked Cross Attention for Image-Text Matching. arXiv.10.1007/978-3-030-01225-0_13 ↗
33. Wang, T., Xu, X., Yang, Y., Hanjalic, A., Shen, H.T., and Song, J. (2019, January 21–25). Matching Images and Text with Multi-Modal Tensor Fusion and Re-Ranking. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350875 ↗
34. Zheng, F., Li, W., Wang, X., Wang, L., Zhang, X., and Zhang, H. (2022). A Cross-Attention Mechanism Based on Regional-Level Semantic Features of Images for Cross-Modal Text-Image Retrieval in Remote Sensing. Appl. Sci., 12.10.3390/app122312221 ↗