A Lightweight Single-Image Super-Resolution Method Based on the Parallel Connection of Convolution and Swin Transformer Blocks
Source abstract
Abstract
In recent years, with the development of deep learning technologies, Vision Transformers combined with Convolutional Neural Networks (CNNs) have made significant progress in the field of single-image super-resolution (SISR). However, existing methods still face issues such as incomplete high-frequency information reconstruction, training instability caused by residual connections, and insufficient cross-window information exchange. To address these problems and better leverage both local and global information, this paper proposes a super-resolution reconstruction network based on the Parallel Connection of Convolution and Swin Transformer Block (PCCSTB) to model the local and global features of an image. Specifically, through a parallel structure of channel feature-enhanced convolution and Swin Transformer, the network extracts, enhances, and fuses the local and global information. Additionally, this paper designs a fusion module to integrate the global and local information extracted by CNNs. The experimental results show that the proposed network effectively balances SR performance and network complexity, achieving good results in the lightweight SR domain. For instance, in the 4× super-resolution experiment on the Urban100 dataset, the network achieves an inference speed of 55 frames per second under the same device conditions, which is more than seven times as fast as the state-of-the-art network Shifted Window-based Image Restoration (SwinIR). Moreover, the network’s Peak Signal-to-Noise Ratio (PSNR) outperforms SwinIR by 0.29 dB at a 4× scale on the Set5 dataset, indicating that the network efficiently performs high-resolution image reconstruction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Optical Systems and Laser Technology · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650000, China
References (34)
Show source-provided references
1. Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6–12). Learning a deep convolutional network for image super-resolution. Proceedings of the 2014 European Conference on Computer Vision, LNCS 8692, Zurich, Switzerland.10.1007/978-3-319-10593-2_13 ↗
2. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
3. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
4. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced deep residual networks for single image super-resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
5. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
7. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the 2020 detection with transformers European Conference on Computer Vision LNCS 12346, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
8. Strudel, R., Garcia, R., Laptev, I., and Schmid, C. (2021, January 11–17). Segmenter: Transformer for semantic segmentation. Proceedings of the 2021 1EEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00717 ↗
9. Li, Y., Zhang, K., Cao, J., Timofte, R., and Van Gool, L.L. (2021). Bringing locality to vision transformers. arXiv.
10. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
11. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. (2021, January 11–17). SwinlR: Image restoration using Swin Transformer. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00210 ↗
12. Chen, Z., Zhang, Y., Gu, J., Kong, L., and Yuan, X. (December, January 28). Cross aggregation Transformer for image restoration. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA.
13. Chen, X., Wang, X., Zhou, J., Qiao, Y., and Dong, C. (2023, January 18–22). Activating more pixels in image super-resolution Transformer. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02142 ↗
14. Tan, M.X., and Le, O.V. (2019, January 9–15). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
15. Wang, H., Chen, X., Ni, B., Liu, Y., and Liu, J. (2023, January 18–22). Omni aggregation networks for lightweight image super-resolution. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02143 ↗
16. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
17. Image Super-Resolution Reconstruction Based on Lightweight Multi-Scale Channel Attention Network
18. Yu, F., and Koltun, V. (2016). Multi-scale context aggregation by dilated convolutions. arXiv.
19. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., and Zhang, L. (2017, January 21–26). NTIRE 2017 challenge on single image super-resolution: Methods and results. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.150 ↗
20. Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi-Morel, M.L. (2012, January 3–7). Low-complexity single-image super-resolution based on nonnegative neighbor embedding. Proceedings of the 2012 British Machine Vision Conference, Guildford, UK.10.5244/C.26.135 ↗
21. Zeyde, R., Elad, M., and Protter, M. (2010, January 24–30). On single image scale-up using sparse-representations. Proceedings of the 2010 International Conference on Curves and Surfaces INCS 6920, Avignon, France.
22. Contour detection and hierarchical image segmentation10.1109/TPAMI.2010.161 ↗
23. Huang, J.B., Singh, A., and Ahuja, N. (2015, January 7–12). Single image super-resolution from transformed self-exemplars. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299156 ↗
24. Sketch-based manga retrieval using Manga109 dataset10.1007/s11042-016-4020-z ↗
25. Image quality assessment based on multiscale geometric analysis10.1109/TIP.2009.2018014 ↗
26. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
27. Kingma, D.P., and Ba, J.L. (2014). Adam: A method for stochastic optimization. arXiv.
28. Ahn, N., Kang, B., and Sohn, K.A. (2018, January 8–14). Fast, accurate, and lightweight super-resolution with cascading residual network. Proceedings of the 15th European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01249-6_16 ↗
29. Li, A., Zhang, L., Liu, Y., and Zhu, C. (2023, January 1–6). Feature modulation Transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01150 ↗
30. Zhang, X., Zeng, H., Guo, S., and Zhang, L. (2022, January 23–27). Efficient long-range attention network for image super-resolution. Proceedings of the 17th European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19790-1_39 ↗
31. Hui, Z., Gao, X., Yang, Y., and Wang, X. (2019, January 21–25). Lightweight image super-resolution with information multi-distillation network. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351084 ↗
32. Li, W., Zhou, K., Qi, L., Jiang, N., Lu, J., and Jia, J. (2020, January 6–12). LAPAR: Linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
33. Luo, X., Xie, Y., Zhang, Y., Qu, Y., Li, C., and Fu, Y. (2020, January 23–28). LatticeNet: Towards lightweight image super-resolution with lattice block. Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58542-6_17 ↗
34. Muqeet, A., Hwang, J., Yang, S., Kang, J., Kim, Y., and Bae, S.H. (2020, January 23–28). Multi-attention based ultra lightweight image super-resolution. Proceedings of the 2020 European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-67070-2_6 ↗