A Prediction-to-Prediction Remote Sensing Image Super-Resolution Network under a Multi-Level Supervision Paradigm
Source abstract
Abstract
Super-resolution enhances the spatial resolution of remote sensing images, yielding clearer data for diverse satellite applications. However, existing methods often lose true detail and produce pseudo-detail in reconstructed images due to an insufficient number of ground truth images for supervision. To address this issue, a prediction-to-prediction super-resolution (P2P-SR) network under a multi-level supervision paradigm was proposed. First, a multi-level supervision network structure was proposed to increase the number of supervisions by introducing more ground truth images, which made the network always predict the next level based on the super-resolution reconstruction results of the previous level. Second, a super-resolution component combining a convolutional neural network and Transformer was designed with a flexible super-resolution scale factor to facilitate the construction of multi-level supervision networks. Finally, a method of dividing the super-resolution overall scale factor was proposed, enabling an investigation into the impact of diverse numbers of components and different scale factors of components on the performance of the multi-level supervision network. Additionally, a new remote sensing dataset containing worldwide scenes was also constructed for the super-resolution task in this paper. The experiment results on three datasets demonstrated that our P2P-SR network outperformed the state-of-the-art (SOTA) methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Advanced Image Fusion Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
Funding
National Natural Science Foundation of China
References (48)
Show source-provided references
1. Pedestrian detection with super-resolution reconstruction for low-quality image10.1016/j.patcog.2021.107846 ↗
2. Wang, L., Li, D., Zhu, Y., Tian, L., and Shan, Y. (2020, January 13–19). Dual super-resolution learning for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00383 ↗
3. Segmentation of tongue muscles from super-resolution magnetic resonance images10.1016/j.media.2014.11.006 ↗
4. Shermeyer, J., and Van Etten, A. (2019, January 15–20). The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00184 ↗
5. Ship detection in spaceborne optical image with SVD networks10.1109/TGRS.2016.2572736 ↗
6. Chen, Y., Liu, S., and Wang, X. (2021, January 20–25). Learning continuous image representation with local implicit image function. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00852 ↗
7. Lei, S., Shi, Z., Wu, X., Pan, B., Xu, X., and Hao, H. (August, January 28). Simultaneous super-resolution and segmentation for remote sensing images. Proceedings of the IGARSS 2019—2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan.10.1109/IGARSS.2019.8900402 ↗
8. Remote sensing image super-resolution and object detection: Benchmark and state of the art10.1016/j.eswa.2022.116793 ↗
9. Remote sensing technology for mapping and monitoring land-cover and land-use change10.1016/S0305-9006(03)00066-7 ↗
10. Li, J., Pei, Y., Zhao, S., Xiao, R., Sang, X., and Zhang, C. (2020). A review of remote sensing for environmental monitoring in China. Remote Sens., 12.10.3390/rs12071130 ↗
11. A review of the status of satellite remote sensing and image processing techniques for mapping natural hazards and disasters10.1177/0309133309339563 ↗
12. A deep journey into super-resolution: A survey10.1145/3390462 ↗
13. Deep learning for image super-resolution: A survey10.1109/TPAMI.2020.2982166 ↗
14. Real-world single image super-resolution: A brief review10.1016/j.inffus.2021.09.005 ↗
15. Single-frame super-resolution in remote sensing: A practical overview10.1080/01431161.2016.1264027 ↗
16. A comprehensive review on deep learning based remote sensing image super-resolution methods10.1016/j.earscirev.2022.104110 ↗
17. Kim, J., Lee, J.K., and Lee, K.M. (July, January 26). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
18. Lugmayr, A., Danelljan, M., Van Gool, L., and Timofte, R. (2020). Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020,, Springer. Proceedings, Part V 16.
19. Kim, Y., and Son, D. (2021, January 20–25). Noise conditional flow model for learning the super-resolution space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00053 ↗
20. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., and Tan, M. (2020, January 13–19). Closed-loop matters: Dual regression networks for single image super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00545 ↗
21. Dong, C., Loy, C.C., He, K., and Tang, X. (2014). Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014, Springer. Proceedings, Part IV 13.
22. Mao, X.J., Shen, C., and Yang, Y.B. (2016). Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv.
23. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (July, January 26). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
24. Dong, C., Loy, C.C., and Tang, X. (2016). Proceedings of the European Conference on Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Springer. Proceedings, Part II 14.
25. Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., and Gao, W. (2021, January 20–25). Pre-Trained Image Processing Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01212 ↗
26. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. (2021, January 11–17). SwinIR: Image Restoration Using Swin Transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00210 ↗
27. Lu, Z., Li, J., Liu, H., Huang, C., Zhang, L., and Zeng, T. (2022, January 18–24). Transformer for Single Image Super-Resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00061 ↗
28. Single-image super resolution for multispectral remote sensing data using convolutional neural networks10.5194/isprs-archives-XLI-B3-883-2016 ↗
29. Single-frame super resolution of remote-sensing images by convolutional neural networks10.1080/01431161.2018.1425561 ↗
30. Super-resolution for remote sensing images via local–global combined network10.1109/LGRS.2017.2704122 ↗
31. Jiang, K., Wang, Z., Yi, P., Jiang, J., Xiao, J., and Yao, Y. (2018). Deep distillation recursive network for remote sensing imagery super-resolution. Remote Sens., 10.10.3390/rs10111700 ↗
32. Fusformer: A transformer-based fusion network for hyperspectral image super-resolution
33. SWCGAN: Generative adversarial network combining swin transformer and CNN for remote sensing image super-resolution10.1109/JSTARS.2022.3190322 ↗
34. TR-MISR: Multiimage super-resolution based on feature fusion with transformers10.1109/JSTARS.2022.3143532 ↗
35. Transformer-Based Multistage Enhancement for Remote Sensing Image Super-Resolution
36. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
37. Ahn, N., Kang, B., and Sohn, K.A. (2018, January 18–23). Image super-resolution via progressive cascading residual network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00123 ↗
38. Nathan, S., and Kansal, P. (2021, January 20–25). Leveraging multi scale backbone with multilevel supervision for thermal image super resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00489 ↗
39. Johnson, J., Alahi, A., and Fei-Fei, L. (2016). Proceedings of the European Conference on Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Springer. Proceedings, Part II 14.
40. Automatic sparse connectivity learning for neural networks10.1109/TNNLS.2022.3141665 ↗
41. Weight-quantized squeezenet for resource-constrained robot vacuums for indoor obstacle classification10.3390/ai3010011 ↗
42. Hu, W., Che, Z., Liu, N., Li, M., Tang, J., Zhang, C., and Wang, J. (2023). Channel Pruning via Class-Aware Trace Ratio Optimization. IEEE Trans. Neural Netw. Learn. Syst.10.1109/TNNLS.2023.3262952 ↗
43. Hsia, C.H., Lee, Y.H., and Lai, C.F. (2022). An Explainable and Lightweight Deep Convolutional Neural Network for Quality Detection of Green Coffee Beans. Appl. Sci., 12.10.20944/preprints202209.0190.v1 ↗
44. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
45. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
46. Image super-resolution with an enhanced group convolutional neural network10.1016/j.neunet.2022.06.009 ↗
47. Yang, Y., and Newsam, S. (2010, January 2–5). Bag-of-visual-words and spatial extensions for land-use classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.10.1145/1869790.1869829 ↗
48. AID: A benchmark data set for performance evaluation of aerial scene classification10.1109/TGRS.2017.2685945 ↗