A Survey on Deep Learning-Driven Remote Sensing Image Scene Understanding: Scene Classification, Scene Retrieval and Scene-Guided Object Detection
Source abstract
Abstract
As a fundamental and important task in remote sensing, remote sensing image scene understanding (RSISU) has attracted tremendous research interest in recent years. RSISU includes the following sub-tasks: remote sensing image scene classification, remote sensing image scene retrieval, and scene-driven remote sensing image object detection. Although these sub-tasks have different goals, they share some communal hints. Hence, this paper tries to discuss them as a whole. Similar to other domains (e.g., speech recognition and natural image recognition), deep learning has also become the state-of-the-art technique in RSISU. To facilitate the sustainable progress of RSISU, this paper presents a comprehensive review of deep-learning-based RSISU methods, and points out some future research directions and potential applications of RSISU.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote-Sensing Image Classification · Advanced Image and Video Retrieval Techniques · Remote Sensing and Land Use
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China
Funding
National Natural Science Foundation of China · China Postdoctoral Science Foundation · Hubei Provincial Natural Science Foundation of China
References (100+)
Show source-provided references
1. Liang, M., and Hu, X. (2015, January 7–12). Recurrent convolutional neural network for object recognition. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
2. Remote Sensing Image Scene Classification: Benchmark and State of the Art10.1109/JPROC.2017.2675998 ↗
3. Xia, G.S., Tong, X.Y., Hu, F., Zhong, Y., Datcu, M., and Zhang, L. (2017). Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation. arXiv.
4. A Survey on Object Detection in Optical Remote Sensing Images10.1016/j.isprsjprs.2016.03.014 ↗
5. Pyramid of Spatial Relatons for Scene-Level Land Use Classification10.1109/TGRS.2014.2351395 ↗
6. Yang, Y., and Newsam, S. (2010, January 2–5). Bag-of-visual-words and spatial extensions for land-use classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.10.1145/1869790.1869829 ↗
7. A Linear Dirichlet Mixture Model for decomposing scenes: Application to analyzing urban functional zonings10.1016/j.rse.2015.07.017 ↗
8. Reducing the Dimensionality of Data with Neural Networks10.1126/science.1127647 ↗
9. Liu, J., Shah, M., Kuipers, B., and Savarese, S. (2011, January 20–25). Cross-view action recognition via view knowledge transfer. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995729 ↗
10. Zafar, B., Ashraf, R., Ali, N., Iqbal, M.K., Sajid, M., Dar, S.H., and Ratyal, N.I. (2018). A Novel Discriminating and Relative Global Spatial Image Representation with Applications in CBIR. Appl. Sci., 8.10.3390/app8112242 ↗
11. Fusion of local and global features for effective image extraction10.1007/s10489-017-0916-1 ↗
12. Object Detection in Optical Remote Sensing Images Based on Weakly Supervised Learning and High-Level Feature Learning10.1109/TGRS.2014.2374218 ↗
13. Semantics-preserving bag-of-words models and applications10.1109/TIP.2010.2045169 ↗
14. Least Squares Support Vector Machine Classifiers10.1023/A:1018628609742 ↗
15. Random Forests for land cover classification10.1016/j.patrec.2005.08.011 ↗
16. Bootstrap resampling for image registration uncertainty estimation without ground truth10.1109/TIP.2009.2030955 ↗
17. Boykov, Y., Veksler, O., and Zabih, R. (1997). Markov Random Fields with Efficient Approximations, Cornell University.
18. Unsupervised Deep Feature Extraction for Remote Sensing Image Classification10.1109/TGRS.2015.2478379 ↗
19. Xia, G.-S., Yang, W., Delon, J., Gousseau, Y., Sun, H., and Maître, H. (2010, January 5). Structural High-resolution Satellite Image Indexing. Proceedings of the ISPRS TC VII Symposium—100 Years ISPRS, Vienna, Austria.
20. Dirichlet-Derived Multiple Topic Scene Classification Model for High Spatial Resolution Remote Sensing Imagery10.1109/TGRS.2015.2496185 ↗
21. Learning a Discriminative Distance Metric With Label Consistency for Scene Classification10.1109/TGRS.2017.2692280 ↗
22. PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval10.1016/j.isprsjprs.2018.01.004 ↗
23. Li, H., Tao, C., Wu, Z., Chen, J., Gong, J., and Deng, M. (2017). RSI-CB: A Large Scale Remote Sensing Image Classification Benchmark via Crowdsource Data.
24. Pu, J., Xia, G.S., Fan, H., Lu, Q., and Zhang, L. (2018, January 22–27). AID++: An Updated Version of AID on Scene Classification. Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.
25. Gao, F., Huang, T., Wang, J., Sun, J., Hussain, A., and Yang, E. (2017). Dual-Branch Deep Convolution Neural Network for Polarimetric SAR Image Classification. Appl. Sci., 7.10.3390/app7050447 ↗
26. Penatti, O.A.B., Nogueira, K., and Santos, J.A.d. (2015, January 7–12). Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA.10.1109/CVPRW.2015.7301382 ↗
27. An, T., and Kim, M. (2010, January 23–24). A New Diverse AdaBoost Classifier. Proceedings of the 2010 International Conference on Artificial Intelligence and Computational Intelligence, Sanya, China.10.1109/AICI.2010.82 ↗
28. A fuzzy K-nearest neighbor algorithm10.1109/TSMC.1985.6313426 ↗
29. Xuming, H., Zemel, R.S., and Carreira-Perpinan, M.A. (July, January 27). Multiscale conditional random fields for image labeling. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, Washington, DC, USA.
30. Kernel sparse representation based classification10.1016/j.neucom.2011.08.018 ↗
31. Classification of heart rate data using artificial neural network and fuzzy equivalence relation10.1016/S0031-3203(02)00063-8 ↗
32. A Linear Support Higher-Order Tensor Machine for Classification10.1109/TIP.2013.2253485 ↗
33. Stacked Autoencoder-based deep learning for remote-sensing image classification: A case study of African land-cover mapping10.1080/01431161.2016.1246775 ↗
34. Domain Adaptation Network for Cross-Scene Classification10.1109/TGRS.2017.2692281 ↗
35. Lan, Z., and Yang, L. (2018). Study on Multi-Scale Window Determination for GLCM Texture Description in High-Resolution Remote Sensing Image Geo-Analysis Supported by GIS and Domain Knowledge. ISPRS Int. J. Geo-Inf., 7.10.3390/ijgi7050175 ↗
36. High-resolution remote sensing image segmentation based on improved RIU-LBP and SRM10.1186/1687-1499-2013-263 ↗
37. Cheng, G., Zhou, P., Yao, X., Yao, C., Zhang, Y., and Han, J. (2016, January 4–6). Object detection in VHR optical remote sensing images via learning rotation-invariant HOG feature. Proceedings of the 2016 4th International Workshop on Earth Observation and Remote Sensing Applications (EORSA), Guangzhou, China.10.1109/EORSA.2016.7552845 ↗
38. Crater Detection Based on Gist Features10.1109/JSTARS.2014.2375066 ↗
39. Cheng, G., Ma, C., Zhou, P., Yao, X., and Han, J. (2016, January 10–15). Scene classification of high resolution remote sensing images using convolutional neural networks. Proceedings of the 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China.10.1109/IGARSS.2016.7729193 ↗
40. When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNs10.1109/TGRS.2017.2783902 ↗
41. Scene Classification via a Gradient Boosting Random Convolutional Network Framework10.1109/TGRS.2015.2488681 ↗
42. Deep Feature Fusion for VHR Remote Sensing Scene Classification10.1109/TGRS.2017.2700322 ↗
43. Zang, M., Wen, D., Liu, T., Zou, H., and Liu, C. (2019). A Fast Sparse Coding Method for Image Classification. Appl. Sci., 9.10.3390/app9030505 ↗
44. Sparse Representation for Computer Vision and Pattern Recognition10.1109/JPROC.2010.2044470 ↗
45. Unsupervised Multilayer Feature Learning for Satellite Image Scene Classification10.1109/LGRS.2015.2503142 ↗
46. Yu, Y., Zhong, P., and Gong, Z. (2017, January 23–28). Balanced data driven sparsity for unsupervised deep feature learning in remote sensing images classification. Proceedings of the 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Fort Worth, TX, USA.10.1109/IGARSS.2017.8127041 ↗
47. Patch-Sorted Deep Feature Learning for High Resolution SAR Image Classification10.1109/JSTARS.2018.2851023 ↗
48. Deep learning in remote sensing scene classification: A data augmentation enhanced convolutional neural network framework10.1080/15481603.2017.1323377 ↗
49. Song, H.O., Xiang, Y., Jegelka, S., and Savarese, S. (2016, January 27–30). Deep Metric Learning via Lifted Structured Feature Embedding. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.434 ↗
50. Duplex Metric Learning for Image Set Classification10.1109/TIP.2017.2760512 ↗
51. A Unified Metric Learning-Based Framework for Co-Saliency Detection10.1109/TCSVT.2017.2706264 ↗
52. Diversity-Promoting Deep Structural Metric Learning for Remote Sensing Scene Classification10.1109/TGRS.2017.2748120 ↗
53. Tang, X., Zhang, X., Liu, F., and Jiao, L. (2018). Unsupervised Deep Feature Learning for Remote Sensing Image Retrieval. Remote Sens., 10.10.3390/rs10081243 ↗
54. Recursive Autoencoders-Based Unsupervised Feature Learning for Hyperspectral Image Classification10.1109/LGRS.2017.2737823 ↗
55. Zhou, W., and Li, C. (2016). Deep feature representations for high-resolution remote-sensing imagery retrieval. Remote Sens., 9.10.3390/rs9050489 ↗
56. Qian, B., and Ping, G. (2004, January 10–13). Comparative studies on similarity measures for remote sensing image retrieval. Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), The Hague, The Netherlands.
57. High-resolution remote-sensing imagery retrieval using sparse features by auto-encoder10.1080/2150704X.2015.1074756 ↗
58. Boualleg, Y., and Farah, M. (2018, January 22–27). Enhanced Interactive Remote Sensing Image Retrieval with Scene Classification Convolutional Neural Networks Model. Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.10.1109/IGARSS.2018.8518388 ↗
59. Zhou, W., Newsam, S., Li, C., and Shao, Z. (2017). Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval. Remote Sens., 9.10.3390/rs9050489 ↗
60. Visual descriptors for content-based retrieval of remote-sensing images
61. Remote Sensing Image Retrieval Using Convolutional Neural Network Features and Weighted Distance10.1109/LGRS.2018.2847303 ↗
62. Hu, F., Tong, X., Xia, G., and Zhang, L. (2016, January 6–10). Delving into deep representations for remote sensing image retrieval. Proceedings of the 2016 IEEE 13th International Conference on Signal Processing (ICSP), Chengdu, China.10.1109/ICSP.2016.7877823 ↗
63. Yandex, A.B., and Lempitsky, V. (2015, January 7–13). Aggregating Local Deep Features for Image Retrieval. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.150 ↗
64. Zhou, W., Deng, X., and Shao, Z. (2018). Region Convolutional Features for Multi-Label Remote Sensing Image Retrieval. arXiv.
65. Ben Ahmed, O., Urruty, T., Richard, N., and Christine, F.-M. (2019). Toward Content-Based Hyperspectral Remote Sensing Image Retrieval (CB-HRSIR): A Preliminary Study Based on Spectral Sensitivity Functions. Remote Sens., 11.10.3390/rs11050600 ↗
66. Li, Y., Zhang, Y., Tao, C., and Zhu, H. (2016). Content-Based High-Resolution Remote Sensing Image Retrieval via Unsupervised Feature Learning and Collaborative Affinity Metric Fusion. Remote Sens., 8.10.3390/rs8090709 ↗
67. Remote Sensing Image Retrieval by Scene Semantic Matching10.1109/TGRS.2012.2217397 ↗
68. Aksoy, S. (2006, January 17). Modeling of Remote Sensing Image Content Using Attributed Relational Graphs. Proceedings of the Structural, Syntactic, and Statistical Pattern Recognition, Berlin, Heidelberg.10.1007/11815921_52 ↗
69. Region-Based Retrieval of Remote Sensing Images Using an Unsupervised Graph-Theoretic Approach10.1109/LGRS.2016.2558289 ↗
70. Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic Method10.1109/TGRS.2017.2760909 ↗
71. A Three-Layered Graph-Based Learning Approach for Remote Sensing Image Retrieval10.1109/TGRS.2016.2579648 ↗
72. Two-Stage Reranking for Remote Sensing Image Retrieval10.1109/TGRS.2017.2714676 ↗
73. Xiao, Z., Long, Y., Li, D., Wei, C., Tang, G., and Liu, J. (2017). High-Resolution Remote Sensing Image Retrieval Based on CNNs from a Dimensional Perspective. Remote Sens., 9.10.3390/rs9070725 ↗
74. Ye, D., Li, Y., Tao, C., Xie, X., and Wang, X. (2017). Multiple Feature Hashing Learning for Large-Scale Remote Sensing Image Retrieval. ISPRS Int. J. Geo-Inf., 6.10.3390/ijgi6110364 ↗
75. Hashing-Based Scalable Remote Sensing Image Search and Retrieval in Large Archives10.1109/TGRS.2015.2469138 ↗
76. Partial Randomness Hashing for Large-Scale Remote Sensing Image Retrieval10.1109/LGRS.2017.2651056 ↗
77. Roy, S., Sangineto, E., Demir, B., and Sebe, N. (2018, January 22–27). Deep Metric and Hash-Code Learning for Content-Based Retrieval of Remote Sensing Images. Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.10.1109/IGARSS.2018.8518381 ↗
78. Chen, C., Zou, H., Shao, N., Sun, J., and Qin, X. (2018, January 22–27). Deep Semantic Hashing Retrieval of Remotec Sensing Images. Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.10.1109/IGARSS.2018.8519276 ↗
79. Zhang, J., Chen, L., Zhuo, L., Liang, X., and Li, J. (2018). An Efficient Hyperspectral Image Retrieval Method: Deep Spectral-Spatial Feature Extraction with DCGAN and Dimensionality Reduction Using t-SNE-Based NM Hashing. Remote Sens., 10.10.3390/rs10020271 ↗
80. Chen, Y., Crawford, M.M., and Ghosh, J. (2005, January 29–29). Applying nonlinear manifold learning to hyperspectral data for land cover classification. Proceedings of the 2005 IEEE International Geoscience and Remote Sensing Symposium, 2005 IGARSS’05, Seoul, Korea.
81. Large-Scale Remote Sensing Image Retrieval by Deep Hashing Neural Networks10.1109/TGRS.2017.2756911 ↗
82. Learning Source-Invariant Deep Hashing Convolutional Neural Networks for Cross-Source Remote Sensing Image Retrieval10.1109/TGRS.2018.2839705 ↗
83. Cao, Y., Niu, X., and Dou, Y. (2016, January 13–15). Region-based convolutional neural networks for object detection in very high resolution remote sensing images. Proceedings of the 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, China.10.1109/FSKD.2016.7603232 ↗
84. Learning Multiscale Deep Features for High-Resolution Satellite Image Scene Classification10.1109/TGRS.2017.2743243 ↗
85. Small Object Detection in Optical Remote Sensing Images via Modified Faster R-CNN10.3390/app8050813 ↗
86. Research for the SSD-Based Storage Technology of Mass Remote Sensing Image
87. A spectral–structural bag-of-features scene classifier for very high spatial resolution remote sensing imagery10.1016/j.isprsjprs.2016.03.004 ↗
88. Built-Up Area Detection From Satellite Images Using Multikernel Learning, Multifield Integrating, and Multihypothesis Voting
89. Unsupervised Deep Feature Learning for Urban Village Detection from High-Resolution Remote Sensing Images10.14358/PERS.83.8.567 ↗
90. Automatic Extraction of Built-Up Areas from Panchromatic and Multispectral Remote Sensing Images Using Double-Stream Deep Convolutional Neural Networks10.1109/JSTARS.2018.2871046 ↗
91. Weakly Supervised Learning for Target Detection in Remote Sensing Images10.1109/LGRS.2014.2358994 ↗
92. Deep networks under scene-level supervision for multi-class geospatial object detection from remote sensing images10.1016/j.isprsjprs.2018.09.014 ↗
93. Learning multiscale and deep representations for classifying remotely sensed imagery10.1016/j.isprsjprs.2016.01.004 ↗
94. Classification of multispectral remote sensing data using a back-propagation neural network10.1109/36.124218 ↗
95. AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification
96. Can a Machine Generate Humanlike Language Descriptions for a Remote Sensing Image?10.1109/TGRS.2017.2677464 ↗
97. Greedy Annotation of Remote Sensing Image Scenes Based on Automatic Aggregation via Hierarchical Similarity Diffusion10.1109/ACCESS.2018.2873761 ↗
98. Mapping regional economic activity from night-time light satellite imagery10.1016/j.ecolecon.2005.03.007 ↗
99. Combining satellite imagery and machine learning to predict poverty10.1126/science.aaf7894 ↗
100. A cluster-based method to map urban area from DMSP/OLS nightlights10.1016/j.rse.2014.03.004 ↗