Image Deblurring Based on an Improved CNN-Transformer Combination Network
Source abstract
Abstract
Recently, using a CNN has been a common practice to restore blurry images due to its strong ability to learn feature information from large-scale datasets. However, CNNs essentially belong to local operations and have the defect of a limited receptive field, which reduces the naturalness of deblurring results. Moreover, CNN-based deblurring methods usually adopt many downsample operations, which hinder detail recovery. Fortunately, transformers focus on modeling the global features, so they can cooperate with CNNs to enlarge the receptive field and compensate for the details lost as well. In this paper, we propose an improved CNN-transformer combination network for deblurring, which adopts a coarse-to-fine architecture as the backbone. To extract the local features and global features simultaneously, the common methods are two blocks connected in parallel or cascaded. Different from these, we design a local-global feature combination block (LGFCB) with a new connecting structure to better use the extracted features. The LGFCB comprises multi-scale residual blocks (MRB) and a transformer block. In addition, we adopt a channel attention fusion block (CAFB) in the encoder path to integrate features. To improve the ability of feature representation, in the decoder path, we introduce a supervised attention block (SAB) operated on restoration images to refine features. Numerous experiments on GoPro and RealBlur datasets indicated that our model achieves remarkable accuracy and processing speed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Digital Media Forensic Detection · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
Funding
Jilin Province Science and Technology Department
References (33)
Show source-provided references
1. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., and Freeman, W.T. (August, January 30). Removing camera shake a single photograph. Proceedings of the ACM SIGGRAPH 2006 Papers, Boston, MA, USA.10.1145/1179352.1141956 ↗
2. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., and Shao, L. (2021, January 19–25). Multi-stage progressive image restoration. Proceedings of the CVPR, Online.10.1109/CVPR46437.2021.01458 ↗
3. Chakrabarti, A. (2016, January 8–16). A neural approach to blind motion deblurring. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_14 ↗
4. Hradiš, M., Kotera, J., Zemčík, P., and Šroubek, F. (2015, January 7–10). Convolutional neural networks for direct text deblurring. Proceedings of the British Machine Vision Conference, Swansea, UK.10.5244/C.29.6 ↗
5. Learning to deblur10.1109/TPAMI.2015.2481418 ↗
6. Sun, J., Cao, W., Xu, Z., and Ponce, J. (2015, January 7–12). Learning a convolutional neural network for non-uniform motion blur removal. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298677 ↗
7. Nah, S., Kim, T.H., and Lee, K.M. (2017, January 21–26). Deep multi-scale convolutional neural network for dynamic scene deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.35 ↗
8. Cho, S.-J., Ji, S.-W., Hong, J.-P., Jung, S.-W., and Ko, S.-J. (2021, January 11–17). Rethinking coarse-to-fine approach in single image deblurring. Proceedings of the ICCV, Montreal, QC, Canada.10.1109/ICCV48922.2021.00460 ↗
9. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., and Matas, J. (2018, January 18–22). Deblurgan: Blind motion deblurring using conditional adversarial networks. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00854 ↗
10. Kupyn, O., Martyniuk, T., Wu, J., and Wang, Z. (November, January 27). Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. Proceedings of the ICCV, Seoul, Republic of Korea.
11. Zhang, J., Pan, J., Ren, S.J., Song, Y., Bao, L., Lau, W.H., and Yang, M.-H. (2018, January 18–22). Dynamic scene deblurring using spatially variant recurrent neural networks. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00267 ↗
12. Tao, X., Gao, H., Shen, X., Wang, J., and Jia, J. (2018, January 18–22). Scale-recurrent network for deep image deblurring. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00853 ↗
13. Zhang, H., Dai, Y., Li, H., and Koniusz, P. (2019, January 16–20). Deep stacked hierarchical multi-patch network for image deblurring. Proceedings of the CVPR, Long Beach, CA, USA.10.1109/CVPR.2019.00613 ↗
14. Park, D., Kang, D.U., Kim, J., and Chun, S.Y. (2020, January 23–28). Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. Proceedings of the European Conference on Computer Vision, Online.10.1007/978-3-030-58539-6_20 ↗
15. Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., and Li, H. (2020, January 14–19). Deblurring by realistic blurring. Proceedings of the CVPR, Online.10.1109/CVPR42600.2020.00281 ↗
16. Zou, W., Jiang, M., Zhang, Y., Chen, L., Lu, Z., and Wu, Y. (2021, January 11–17). SDWNet: A straight dilated network with wavelet transformation for image deblurring. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada.10.1109/ICCVW54120.2021.00216 ↗
17. BANet: A blur-aware attention network for dynamic scene deblurring10.1109/TIP.2022.3216216 ↗
18. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M.-H. (2022, January 19–24). Restormer: Efficient transformer for high-resolution image restoration. Proceedings of the CVPR, New Orleans, USA.10.1109/CVPR52688.2022.00564 ↗
19. Kumar, J., Mastan, I.D., and Raman, S. (2021). FMD-cGAN: Fast motion deblurring using conditional generative adversarial networks. Communications in Computer and Information Science, Springer. CVIP.10.1007/978-3-031-11349-9_32 ↗
20. Gao, G., Xu, Z., Li, J., Yang, J., Zeng, T., and Qi, G.-J. (2022). CTCNet: A CNN-transformer cooperation network for face image super-resolution. arXiv.10.1109/TIP.2023.3261747 ↗
21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
22. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
23. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Online.
24. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
25. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
26. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
27. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
28. Huang, P., Han, S., Zhao, J., Liu, D., Wang, H., Yu, E., and Kot, A.C. (2020). Refinements in motion and appearance for online multi-object tracking. arXiv.
29. Charbonnier, P., Blanc-Feraud, L., Aubert, G., and Barlaud, M. (1994, January 13–16). Two deterministic half-quadratic regularization algorithms for computed imaging. Proceedings of the ICIP, Austin, TX, USA.
30. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
31. Rim, J., Lee, H., Won, J., and Cho, S. (2020, January 23–28). Real-world blur dataset for learning and benchmarking deblurring algorithms. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58595-2_12 ↗
32. Loshchilov, I., and Hutter, F. (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv.
33. Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y.M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv.