OSM-CLIP: Enhancing Remote Sensing Image–Text Representation Learning with OpenStreetMap Data
Source abstract
Abstract
Remote sensing vision–language models, such as RemoteCLIP and GeoRSCLIP, have advanced image–text representation learning. However, they rely on manually curated caption datasets that are expensive to scale and provide only global image-level supervision. In this paper, we introduce OSM-CLIP, a framework that exploits the freely available, continuously growing annotations of OpenStreetMap (OSM) to provide regionally scalable, patch-level supervision for remote sensing image-text learning. We construct a large-scale dataset of over 265,000 satellite images covering the contiguous United States, each automatically paired with fine-grained geographic annotations scraped from OSM and mapped to individual image patches. A contrastive loss operating at the patch level associates each image region with its corresponding OSM textual description, enabling the model to learn spatially grounded representations without any manual labeling effort. After fine-tuning on standard remote sensing captioning datasets, OSM-CLIP achieves an average improvement of 10.81% in zero-shot classification, 5.06% in text-to-image retrieval (R@1), and 3.87% in image-to-text retrieval (R@1) over existing methods across 13 classification and 4 retrieval benchmarks. Our results demonstrate that freely available geographic annotations can serve as a powerful source of supervision for remote sensing vision–language models in regions with high-quality OSM coverage.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Geographic Information Systems Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy
References (28)
Show source-provided references
1. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual.
2. RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
3. RS5M and GeoRSCLIP: A Large-Scale Vision-Language Dataset and a Large Vision-Language Model for Remote Sensing
4. Chen, W., Chen, J., Deng, Y., Chen, J., Feng, Y., Xi, Z., Liu, D., Li, K., and Meng, Y. (2025). LRSCLIP: A Vision-Language Foundation Model for Aligning Remote Sensing Image with Longer Text. arXiv.
5. RS-CLIP: Zero Shot Remote Sensing Scene Classification via Contrastive Vision-Language Supervision
6. ChangeCLIP: Remote Sensing Change Detection with Multimodal Vision-Language Representation Learning10.1016/j.isprsjprs.2024.01.004 ↗
7. Marimo, C.T., Blumenstiel, B., Nitsche, M., Jakubik, J., and Brunschwiler, T. (2025). Beyond the Visible: Multispectral Vision-Language Learning for Earth Observation. arXiv.10.1007/978-3-032-06106-5_21 ↗
8. Wang, Z., Prabha, R., Huang, T., Wu, J., and Rajagopal, R. (2024, January 20–27). SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing. Proceedings of the 2024 AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
9. Mall, U., Hariharan, B., and Bala, K. (2024, January 7–11). Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
10. Audebert, N., Le Saux, B., and Lefèvre, S. (2017, January 21–26). Joint Learning from Earth Observation and OpenStreetMap Data to Get Faster Better Semantic Maps. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPRW.2017.199 ↗
11. Usmani, M., Bovolo, F., and Napolitano, M. (2023). Remote Sensing and Deep Learning to Understand Noisy OpenStreetMap. Remote Sens., 15.10.3390/rs15184639 ↗
12. Integrating Remote Sensing with OpenStreetMap Data for Comprehensive Scene Understanding through Multi-Modal Self-Supervised Learning10.1016/j.rse.2024.114573 ↗
13. CromSS: Cross-Modal Pre-Training with Noisy Labels for Remote Sensing Image Segmentation
14. Bazi, Y., Bashmal, L., Al Rahhal, M.M., Al Dayil, R., and Al Ajlan, N. (2021). Vision Transformers for Remote Sensing Image Classification. Remote Sens., 13.10.3390/rs13030516 ↗
15. Jiang, W., Zhang, J., Wang, D., Zhang, Q., Wang, Z., and Du, B. (2024, January 3–9). LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, Jeju, Republic of Korea.
16. Advancing Plain Vision Transformer toward Remote Sensing Foundation Model
17. Reed, C.J., Gupta, R., Li, S., Brber, S., Zhu, S., Samber, N., Ebrahimi, S., Vishwanath, V., Kornblith, S., and Darrell, T. (2023, January 2–6). Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning. Proceedings of the International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00378 ↗
18. Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., and Ermon, S. (December, January 28). SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery. Proceedings of the Advances in Neural Information Processing Systems 35 (NeurIPS), New Orleans, LA, USA.
19. Yun, S., Kim, S., Han, D., Song, H., Song, J., and Hong, Y. (2022, January 18–24). Patch-Level Representation Learning for Self-Supervised Vision Transformers. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00817 ↗
20. Jean, N., Wang, S., Samar, A., Azzari, G., Lobell, D., and Ermon, S. (February, January 27). Tile2Vec: Unsupervised Representation Learning for Spatially Distributed Data. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), Honolulu, HI, USA.
21. GLCNet: Global-Local Contrastive Network for Self-Supervised Representation Learning in Remote Sensing
22. CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding10.1109/TGRS.2023.3268232 ↗
23. Zhang, J., Wang, D., Liu, Y., Zhang, Q., and Du, B. (2024). Multimodal Contrastive Learning for Remote Sensing Image Feature Extraction Based on Relaxed Positive Samples. Sensors, 24.10.3390/s24237719 ↗
24. Shazeer, N., and Stern, M. (2018, January 10–15). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
25. Remote Sensing Image Scene Classification: Benchmark and State of the Art10.1109/JPROC.2017.2675998 ↗
26. Exploring Models and Data for Remote Sensing Image Caption Generation10.1109/TGRS.2017.2776321 ↗
27. Yang, Y., and Newsam, S. (2010, January 2–5). Bag-of-Visual-Words and Spatial Extensions for Land-Use Classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.10.1145/1869790.1869829 ↗
28. Zhang, X., Wang, T., Qi, J., Lu, H., and Wang, G. (2021). Saliency Detection for Remote Sensing Images: A Survey. Remote Sens., 13.10.3390/rs13214291 ↗