SupGAN: A General Super-Resolution GAN-Promoting Training Method
Source abstract
Abstract
An image super-resolution (SR) method based on Generative Adversarial Networks (GANs) has achieved impressive results in terms of visual performance. However, the weights of loss functions in these methods are usually set to fixed values manually, which cannot fully adapt to different datasets and tasks, and may result in a decrease in the perceptual effect of the SR images. To address this issue and further improve visual quality, we propose a perception-driven SupGAN, which improves the generator and loss function of GAN-based image super-resolution models. The generator adopts multi-scale feature extraction and fusion to restore SR images with diverse and fine textures. We design a network-training method based on the proportion of high-frequency information in images (BHFTM), which utilizes the proportion of high-frequency information in images obtained through the Canny operator to set the weights of the loss function. In addition, we employ the four-patch method to better simulate the degradation of complex real-world scenarios. We extensively test our method and compare it with recent SR methods (BSRGAN, Real-ESRGAN, RealSR, SwinIR, LDL, etc.) on different types of datasets (OST300, 2020track1, RealWorld38, BSDS100 etc.) with a scaling factor of ×4. The results show that the NIQE metric improves, and also demonstrate that SupGAN can generate more natural and fine textures while suppressing unpleasant artifacts.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software Engineering, Huazhong University of Science and Technology, Wuhan 430074, China
School of Journalism and Information Communication, Huazhong University of Science and Technology, Wuhan 430074, China
Funding
Tencent Co.
References (52)
Show source-provided references
1. An edge-guided image interpolation algorithm via directional filtering and data fusion10.1109/TIP.2006.877407 ↗
2. Lanczos filtering in one and two dimensions10.1175/1520-0450(1979)018<1016:LFIOAT>2.0.CO;2 ↗
3. Softcuts: A soft edge smoothness prior for color image super-resolution10.1109/TIP.2009.2012908 ↗
4. Sun, J., Xu, Z., and Shum, H.-Y. (2008, January 23–28). Image super-resolution using gradient profile prior. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.
5. Single image super-resolution based on gradient profile sharpness10.1109/TIP.2015.2414877 ↗
6. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
7. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
8. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., and Loy, C.C. (2018, January 8–14). ESRGAN: Enhanced super-resolution generative adversarial networks. Proceedings of the European Conference on Computer Vision Workshops, Munich, Germany.10.1007/978-3-030-11021-5_5 ↗
9. Jolicoeur-Martineau, A. (2018). The relativistic discriminator: A key element missing from standard gan. arXiv.
10. Wang, X., Xie, L., Dong, C., and Shan, Y. (2021, January 11–17). Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data. Proceedings of the ICCVW, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00217 ↗
11. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate image super-resolution using very deep convolutional networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
12. Lai, W.-S., Huang, J.-B., Ahuja, N., and Yang, M. (2017, January 21–26). Deep laplacian pyramid networks for fast and accurate super-resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.618 ↗
13. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017, January 21–26). Ntire 2017 challenge on single image super-resolution: Methods and results. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.150 ↗
14. Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6–12). Learning a deep convolutional network for image super-resolution. Proceedings of the ECCV, Zurich, Switzerland.10.1007/978-3-319-10593-2_13 ↗
15. Dong, C., Loy, C.C., and Tang, X. (2016, January 11–14). Accelerating the Super-Resolution Convolutional Neural Network. Proceedings of the ECCV, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_25 ↗
16. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
18. Yang, F., Yang, H., Fu, J., Lu, H., and Guo, B. (2020, January 13–19). Learning texture transformer network for image super-resolution. Proceedings of the CVPR, Seattle, WA, USA.10.1109/CVPR42600.2020.00583 ↗
19. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the ECCV, Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
20. Shang, T., Dai, Q., Zhu, S., Yang, T., and Guo, Y. (2020, January 14–19). Perceptual Extreme Super Resolution Network with Receptive Field Block. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.10.1109/CVPRW50498.2020.00228 ↗
21. Liu, S., and Huang, D. (2018, January 8–14). Receptive field block net for accurate and fast object detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01252-6_24 ↗
22. Zhao, H., Gallo, O., Frosio, I., and Kautz, J. (2015). Loss functions for neural networks for image processing. arXiv.
23. Johnson, J., Alahi, A., and Li, F.-F. (2016). Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision, Springer International Publishing.10.1007/978-3-319-46475-6_43 ↗
24. Bruna, J., Sprechmann, P., and LeCun, Y. (2015). Super-resolution with deep convolutional sufficient statistics. arXiv.
25. Sajjadi, M.S.M., Scholkopf, B., and Hirsch, M. (2017, January 22–29). EnhanceNet: Single image super-resolution through automated texture synthesis. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.481 ↗
26. Mechrez, R., Talmi, I., and Zelnik-Manor, L. (2018, January 8–14). The Contextual Loss for Image Transformation with Non-Aligned Data. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_47 ↗
27. Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images10.1109/83.650118 ↗
28. On bayesian adaptive video super resolution10.1109/TPAMI.2013.127 ↗
29. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., and Lin, L. (2018, January 18–23). Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00113 ↗
30. Fritsche, M., Gu, S., and Timofte, R. (2019, January 27–28). Frequency separation for real-world super-resolution. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW.2019.00445 ↗
31. Wang, L., Wang, Y., Dong, X., Xu, Q., Yang, J., An, W., and Guo, Y. (2021, January 20–25). Unsupervised Degradation Representation Learning for Blind Super-Resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01044 ↗
32. Zhang, K., Liang, J., Gool, L.V., and Timofte, R. (2021, January 11–17). Designing a practical degradation model for deep blind image super-resolution. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada.10.1109/ICCV48922.2021.00475 ↗
33. An information fidelity criterion for image quality assessment using natural scene statistics10.1109/TIP.2005.859389 ↗
34. Sheikh, H.R., and Bovik, A.C. (2004, January 17–21). Image information and visual quality. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, QC, Canada.
35. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
36. Learning a noreference quality metric for single-image super-resolution10.1016/j.cviu.2016.12.009 ↗
37. Making a “completely blind” image quality analyzer10.1109/LSP.2012.2227726 ↗
38. No-reference image quality assessment in the spatial domain10.1109/TIP.2012.2214050 ↗
39. Blau, Y., and Michaeli, T. (2018, January 18–22). The perception-distortion tradeoff. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00652 ↗
40. Vasu, S., Madam, N.T., and Rajagopalan, A.N. (2018, January 8–14). Analyzing Perception-Distortion Tradeoff Using Enhanced Perceptual Super-Resolution Network. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany.10.1007/978-3-030-11021-5_8 ↗
41. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., and Zelnik-Manor, L. (2018, January 8–14). The 2018 PIRM Challenge on Perceptual Image Super-Resolution. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany.10.1007/978-3-030-11021-5_21 ↗
42. Christian, S., Wei, L., Yangqing, J., Pierre, S., Scott, R., Dragomir, A., Dumitru, E., Vincent, V., and Andrew, R. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
43. A Computational Approach to Edge Detection
44. Roberts, L.G. (1963). Machine Perception of Three-Dimensional Solids. Optical and Electro-Optical Information Processing. [Doctoral’s Dissertation, Massachusetts Institute of Technology].
45. Freeman, H. (1990). An isotropic 3 × 3 image gradient operator. Machine Vision for Three-Dimensional Scenes, Academic Press.
46. Object enhancement and extraction
47. Kingma, D.P., and Ba, J.L. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the ICLR, San Diego, CA, USA.
48. Agustsson, E., and Timofte, R. (2017, January 21–26). Ntire 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.150 ↗
49. Wang, X., Yu, K., Dong, C., and Loy, C.C. (2018, January 18–22). Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00070 ↗
50. Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 7–14). A database of human segmented naturalimages and its application to evaluating segmentation algorithms and measuringecological statistics. Proceedings of the Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada.
51. Lugmayr, A., Danelljan, M., and Timofte, R. (2020, January 14–19). Ntire 2020 challenge on real-world image super-resolution: Methods and results. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.
52. Huang, J.B., Singh, A., and Ahuja, N. (2015, January 7–12). Single image super-resolution from transformedself-exemplars. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299156 ↗