RSTSRN: Recursive Swin Transformer Super-Resolution Network for Mars Images
Source abstract
Abstract
High-resolution optical images will provide planetary geology researchers with finer and more microscopic image data information. In order to maximize scientific output, it is necessary to further increase the resolution of acquired images, so image super-resolution (SR) reconstruction techniques have become the best choice. Aiming at the problems of large parameter quantity and high computational complexity in current deep learning-based image SR reconstruction methods, we propose a novel Recursive Swin Transformer Super-Resolution Network (RSTSRN) for SR applied to images. The RSTSRN improves upon the LapSRN, which we use as our backbone architecture. A Residual Swin Transformer Block (RSTB) is used for more efficient residual learning, which consists of stacked Swin Transformer Blocks (STBs) with a residual connection. Moreover, the idea of parameter sharing was introduced to reduce the number of parameters, and a multi-scale training strategy was designed to accelerate convergence speed. Experimental results show that the proposed RSTSRN achieves superior performance on 2×, 4× and 8×SR tasks to state-of-the-art methods with similar parameters. Especially on high-magnification SR tasks, the RSTSRN has great performance superiority. Compared to the LapSRN network, for 2×, 4× and 8× Mars image SR tasks, the RSTSRN network has increased PSNR values by 0.35 dB, 0.88 dB and 1.22 dB, and SSIM values by 0.0048, 0.0114 and 0.0311, respectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Image Fusion Techniques · Image and Signal Denoising Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
Key Laboratory of Lunar and Deep Space Exploration, Chinese Academy of Sciences, Beijing 100101, China
Funding
Natural Science Foundation of Jilin Province · National Natural Science Foundation of China · Key Laboratory of Lunar and Deep Space Exploration, Chinese Academy of Sciences
References (85)
Show source-provided references
1. Pancam Multispectral Imaging Results from the Spirit Rover at Gusev Crater10.1126/science.1100175 ↗
2. Pancam Multispectral Imaging Results from the Opportunity Rover at Meridiani Planum10.1126/science.1105245 ↗
3. Curiosity at Gale Crater, Mars: Characterization and Analysis of the Rocknest Sand Shadow10.1126/science.1239505 ↗
4. A Habitable Fluvio-Lacustrine Environment at Yellowknife Bay, Gale Crater, Mars10.1126/science.1242777 ↗
5. Mars Reconnaissance Orbiter’s High Resolution Imaging Science Experiment (HiRISE)10.1029/2005JE002605 ↗
6. Ultrahigh resolution topographic mapping of Mars with MRO HiRISE stereo images: Meter-scale slopes of candidate Phoenix landing sites10.1029/2007JE003000 ↗
7. High Resolution Imaging Science Experiment (HiRISE) images of volcanic terrains from the first 6 months of the Mars Reconnaissance Orbiter Primary Science Phase10.1029/2007JE002968 ↗
8. Observations of periglacial landforms in Utopia Planitia with the High Resolution Imaging Science Experiment (HiRISE)10.1029/2008JE003264 ↗
9. New and recent gully activity on Mars as seen by HiRISE10.1029/2009GL041351 ↗
10. Lai, W.S., Huang, J.B., Ahuja, N., and Yang, M.H. (2017, January 21–26). Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.618 ↗
11. Example-based super-resolution for single-image analysis from the Chang’e-1 Mission10.1088/1674-4527/16/11/172 ↗
12. Super-Resolution Reconstruction for Multi-Angle Remote Sensing Images Considering Resolution Differences10.3390/rs6010637 ↗
13. Tao, Y., and Muller, J.P. (2019). Super-Resolution Restoration of MISR Images Using the UCL MAGiGAN System. Remote Sens., 11.10.3390/rs11010052 ↗
14. SD-GAN: Saliency-Discriminated GAN for Remote Sensing Image Superresolution10.1109/LGRS.2019.2956969 ↗
15. E-DBPN: Enhanced Deep Back-Projection Networks for Remote Sensing Scene Image Superresolution10.1109/TGRS.2020.2966669 ↗
16. Sparse representation-based MRI super-resolution reconstruction10.1016/j.measurement.2013.10.026 ↗
17. Multi-Contrast Super-Resolution MRI through a Progressive Network10.1109/TMI.2020.2974858 ↗
18. Wu, Z., Chen, X., Xie, S., Shen, J., and Zeng, Y. (2023). Super-resolution of brain MRI images based on denoising diffusion probabilistic model. Biomed. Signal Process. Control, 85.10.1016/j.bspc.2023.104901 ↗
19. Eigenface-domain super-resolution for face recognition10.1109/TIP.2003.811513 ↗
20. Face Hallucination Using Cascaded Super-Resolution and Identity Priors10.1109/TIP.2019.2945835 ↗
21. Semi-Cycled Generative Adversarial Networks for Real-World Face Super-Resolution10.1109/TIP.2023.3240845 ↗
22. Joint MAP registration and high-resolution image estimation using a sequence of undersampled images10.1109/83.650116 ↗
23. Fast and Robust Multiframe Super Resolution10.1109/TIP.2004.834669 ↗
24. Multiframe Super-Resolution Employing a Spatially Weighted Total Variation Model10.1109/TCSVT.2011.2163447 ↗
25. Example-based super-resolution10.1109/38.988747 ↗
26. Deep Learning for Image Super-Resolution: A Survey10.1109/TPAMI.2020.2982166 ↗
27. Tao, Y., Conway, S.J., Muller, J.P., Putri, A.R.D., Thomas, N., and Cremonese, G. (2021). Single Image Super-Resolution Restoration of TGO CaSSIS Colour Images: Demonstration with Perseverance Rover Landing Site and Mars Science Targets. Remote Sens., 13.10.3390/rs13091777 ↗
28. High-resolution image recovery from image-plane arrays, using convex projections10.1364/JOSAA.6.001715 ↗
29. Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time10.1109/83.605404 ↗
30. Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants10.1109/83.892456 ↗
31. Improving resolution by image registration10.1016/1049-9652(91)90045-L ↗
32. Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency10.1006/jvci.1993.1030 ↗
33. A Bayesian approach to image expansion for improved definition10.1109/83.287017 ↗
34. Extraction of high-resolution frames from video sequences10.1109/83.503915 ↗
35. A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution10.1109/TIP.2006.888334 ↗
36. Maximum a Posteriori Video Super-Resolution Using a New Multichannel Image Prior10.1109/TIP.2010.2042115 ↗
37. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images10.1109/83.650118 ↗
38. Image Super-Resolution Via Sparse Representation10.1109/TIP.2010.2050625 ↗
39. Tao, Y., Xiong, S., Song, R., and Muller, J.P. (2021). Towards Streamlined Single-Image Super-Resolution: Demonstration with 10 m Sentinel-2 Colour and 10–60 m Multi-Spectral VNIR and SWIR Bands. Remote Sens., 13.10.3390/rs13132614 ↗
40. Tao, Y., Xiong, S., Muller, J.P., Michael, G., Conway, S.J., Paar, G., Cremonese, G., and Thomas, N. (2022). Subpixel-Scale Topography Retrieval of Mars Using Single-Image DTM Estimation and Super-Resolution Restoration. Remote Sens., 14.10.3390/rs14020257 ↗
41. Zou, H., He, S., Cao, X., Sun, L., Wei, J., Liu, S., and Liu, J. (2022). Rescaling-Assisted Super-Resolution for Medium-Low Resolution Remote Sensing Ship Detection. Remote Sens., 14.10.3390/rs14112566 ↗
42. Image Super-Resolution Using Deep Convolutional Networks10.1109/TPAMI.2015.2439281 ↗
43. Dong, C., Loy, C.C., and Tang, X. (2016). Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision—ECCV 2016, Springer International Publishing.10.1007/978-3-319-46475-6_25 ↗
44. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
45. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate Image Super-Resolution Using Very Deep Convolutional Networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
46. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Deeply-Recursive Convolutional Network for Image Super-Resolution. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.181 ↗
47. Wang, Z., Liu, D., Yang, J., Han, W., and Huang, T. (2015, January 7–13). Deep Networks for Image Super-Resolution with Sparse Prior. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.50 ↗
48. Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution10.1109/TIP.2017.2750403 ↗
49. Tai, Y., Yang, J., Liu, X., and Xu, C. (2017, January 22–29). MemNet: A Persistent Memory Network for Image Restoration. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.486 ↗
50. Tai, Y., Yang, J., and Liu, X. (2017, January 21–26). Image Super-Resolution via Deep Recursive Residual Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.298 ↗
51. Tong, T., Li, G., Liu, X., and Gao, Q. (2017, January 22–29). Image Super-Resolution Using Dense Skip Connections. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.514 ↗
52. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
53. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. (2018, January 18–23). Residual Dense Network for Image Super-Resolution. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00262 ↗
54. Liang, J., Cao, J., Sun, G., Zhang, K., Gool, L.V., and Timofte, R. (2021). SwinIR: Image Restoration Using Swin Transformer. arXiv.10.1109/ICCVW54120.2021.00210 ↗
55. Choi, H., Lee, J., and Yang, J. (2023, January 17–24). N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00206 ↗
56. Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., and Yu, F. (2023, January 1–6). Dual Aggregation Transformer for Image Super-Resolution. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01131 ↗
57. Tao, Y., Douté, S., Muller, J.P., Conway, S.J., Thomas, N., and Cremonese, G. (2021). Ultra-High-Resolution 1 m/pixel CaSSIS DTM Using Super-Resolution Restoration and Shape-from-Shading: Demonstration over Oxia Planum on Mars. Remote Sens., 13.10.3390/rs13112185 ↗
58. Tao, Y., and Muller, J.P. (2021). Super-Resolution Restoration of Spaceborne Ultra-High-Resolution Images Using the UCL OpTiGAN System. Remote Sens., 13.10.3390/rs13122269 ↗
59. Enhancing Lunar Reconnaissance Orbiter Images via Multi-Frame Super Resolution for Future Robotic Space Missions10.1109/LRA.2021.3097510 ↗
60. Mars Image Super-Resolution Based on Generative Adversarial Network10.1109/ACCESS.2021.3101858 ↗
61. Tewari, A., Prateek, C., and Khanna, N. (2021). In-Orbit Lunar Satellite Image Super Resolution for Selective Data Transmission. arXiv.
62. An Unsupervised Remote Sensing Single-Image Super-Resolution Method Based on Generative Adversarial Network10.1109/ACCESS.2020.2972300 ↗
63. A Multi-Degradation Aided Method for Unsupervised Remote Sensing Image Super Resolution With Convolution Neural Networks
64. Lightweight Mars remote sensing image super-resolution reconstruction network10.37188/OPE.20223012.1487 ↗
65. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.10.1109/ICCV48922.2021.00986 ↗
66. Agustsson, E., and Timofte, R. (2017, January 21–26). NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.150 ↗
67. Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi Morel, M.-L. (2012, January 3–7). Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. Proceedings of the British Machine Vision Conference 2012, Surrey, UK.10.5244/C.26.135 ↗
68. Zeyde, R., Elad, M., and Protter, M. (2012). On Single Image Scale-Up Using Sparse-Representations. Curves and Surfaces, Springer.10.1007/978-3-642-27413-8_47 ↗
69. Contour Detection and Hierarchical Image Segmentation10.1109/TPAMI.2010.161 ↗
70. Huang, J.B., Singh, A., and Ahuja, N. (2015, January 7–12). Single image super-resolution from transformed self-exemplars. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299156 ↗
71. Sketch-based manga retrieval using manga109 dataset10.1007/s11042-016-4020-z ↗
72. Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017, January 21–26). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
73. Ahn, N., Kang, B., and Sohn, K.A. (2018). Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network. Computer Vision—ECCV 2018, Springer International Publishing.10.1109/CVPRW.2018.00123 ↗
74. Haris, M., Shakhnarovich, G., and Ukita, N. (2018, January 18–23). Deep Back-Projection Networks for Super-Resolution. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00179 ↗
75. Li, J., Fang, F., Mei, K., and Zhang, G. (2018). Multi-scale Residual Network for Image Super-Resolution. Computer Vision—ECCV 2018, Springer International Publishing.10.1007/978-3-030-01237-3_32 ↗
76. Hui, Z., Gao, X., Yang, Y., and Wang, X. (2019, January 21–25). Lightweight Image Super-Resolution with Information Multi-distillation Network. Proceedings of the Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351084 ↗
77. Wang, C., Li, Z., and Shi, J. (2019). Lightweight Image Super-Resolution with Adaptive Weighted Learning Network. arXiv.
78. Liu, J., Tang, J., and Wu, G. (2020). Residual Feature Distillation Network for Lightweight Image Super-Resolution. Computer Vision—ECCV 2020 Workshops, Springer International Publishing.10.1007/978-3-030-67070-2_2 ↗
79. Luo, X., Xie, Y., Zhang, Y., Qu, Y., Li, C., and Fu, Y. (2020). LatticeNet: Towards Lightweight Image Super-Resolution with Lat-tice Block. Computer Vision—ECCV 2020, Springer International Publishing.10.1007/978-3-030-58542-6_17 ↗
80. Fang, J., Lin, H., Chen, X., and Zeng, K. (2022, January 19–20). A Hybrid Network of CNN and Transformer for Lightweight Image Super-Resolution. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA.10.1109/CVPRW56347.2022.00119 ↗
81. Zhang, X., Zeng, H., Guo, S., and Zhang, L. (2022). Efficient Long-Range Attention Network for Image Super-Resolution. Computer Vision—ECCV 2022, Springer Nature.10.1007/978-3-031-19790-1_39 ↗
82. Lu, Z., Li, J., Liu, H., Huang, C., Zhang, L., and Zeng, T. (2022, January 19–20). Transformer for Single Image Super-Resolution. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA.10.1109/CVPRW56347.2022.00061 ↗
83. Zhang, A., Ren, W., Liu, Y., and Cao, X. (2023, January 1–6). Lightweight Image Super-Resolution with Superpixel Token Interaction. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01169 ↗
84. Single image super-resolution based on directional variance attention network10.1016/j.patcog.2022.108997 ↗
85. Image Quality Assessment: From Error Visibility to Structural Similarity10.1109/TIP.2003.819861 ↗