Harnessing Spatial-Frequency Information for Enhanced Image Restoration
Source abstract
Abstract
Image restoration aims to recover high-quality, clear images from those that have suffered visibility loss due to various types of degradation. Numerous deep learning-based approaches for image restoration have shown substantial improvements. However, there are two notable limitations: (a) Despite substantial spectral mismatches in the frequency domain between clean and degraded images, only a few approaches leverage information from the frequency domain. (b) Variants of attention mechanisms have been proposed for high-resolution images in low-level vision tasks, but these methods still require inherently high computational costs. To address these issues, we propose a Frequency-Aware Network (FreANet) for image restoration, which consists of two simple yet effective modules. We utilize a multi-branch/domain module that integrates latent features from the frequency and spatial domains using the discrete Fourier transform (DFT) and complex convolutional neural networks. Furthermore, we introduce a multi-scale pooling attention mechanism that employs average pooling along the row and column axes. We conducted extensive experiments on image restoration tasks, including defocus deblurring, motion deblurring, dehazing, and low-light enhancement. The proposed FreANet demonstrates remarkable results compared to previous approaches to these tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image and Signal Denoising Methods · Photoacoustic and Ultrasonic Imaging · Optical Coherence Tomography Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Intelligent Electronics and Computer Engineering, Chonnam National University, Gwangju 61186, Republic of Korea
Department of Smart ICT Convergence Engineering, Seoul National University of Science and Technology, Seoul 01811, Republic of Korea
School of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea
Funding
Seoul National University of Science and Technology
References (58)
Show source-provided references
1. Single image haze removal using dark channel prior
2. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior10.1109/TIP.2016.2612882 ↗
3. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., and Shao, L. (2021, January 20–25). Multi-stage progressive image restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01458 ↗
4. Ren, W., Ma, L., Zhang, J., Pan, J., Cao, X., Liu, W., and Yang, M.H. (2018, January 18–23). Gated Fusion Network for Single Image Dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00343 ↗
5. Chen, L., Lu, X., Zhang, J., Chu, X., and Chen, C. (2021, January 19–25). HINet: Half Instance Normalization Network for Image Restoration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, TN, USA.10.1109/CVPRW53098.2021.00027 ↗
6. Qin, X., Wang, Z., Bai, Y., Xie, X., and Jia, H. (2020, January 7–12). FFA-Net: Feature fusion attention network for single image dehazing. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6865 ↗
7. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M.H. (2022, January 18–24). Restormer: Efficient transformer for high-resolution image restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00564 ↗
8. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
9. Vision transformers for single image dehazing10.1109/TIP.2023.3256763 ↗
10. Cui, Y., Tao, Y., Bing, Z., Ren, W., Gao, X., Cao, X., Huang, K., and Knoll, A. (2023, January 1–5). Selective frequency network for image restoration. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.
11. Cui, Y., Ren, W., Cao, X., and Knoll, A. (2023, January 1–6). Focal network for image restoration. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01195 ↗
12. Yang, H.H., and Fu, Y. (2019, January 22–25). Wavelet u-net and the chromatic adaptation transform for single image dehazing. Proceedings of the IEEE International Conference on Image Processing, Taipei, Taiwan.10.1109/ICIP.2019.8803391 ↗
13. Li, R., Dong, H., Wang, L., Liang, B., Guo, Y., and Wang, F. (2022, January 21–25). Frequency-aware deep dual-path feature enhancement network for image dehazing. Proceedings of the International Conference on Pattern Recognition, Montreal, QC, Canada.10.1109/ICPR56361.2022.9955635 ↗
14. Zhou, M., Huang, J., Guo, C.L., and Li, C. (2023, January 23–29). Fourmer: An efficient global modeling paradigm for image restoration. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
15. Mao, X., Liu, Y., Liu, F., Li, Q., Shen, W., and Wang, Y. (2023, January 7–14). Intriguing findings of frequency selection for image deblurring. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i2.25281 ↗
16. Guo, M.H., Lu, C.Z., Hou, Q., Liu, Z., Cheng, M.M., and Hu, S.M. (December, January 28). Segnext: Rethinking convolutional attention design for semantic segmentation. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA.
17. Trabelsi, C., Bilaniuk, O., Zhang, Y., Serdyuk, D., Subramanian, S., Santos, J.F., Mehri, S., Rostamzadeh, N., Bengio, Y., and Pal, C.J. (2017). Deep complex networks. arXiv.
18. Cui, Y., Ren, W., Yang, S., Cao, X., and Knoll, A. (2023, January 23–29). IRNeXt: Rethinking Convolutional Network Design for Image Restoration. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
19. Cui, Y., Ren, W., and Knoll, A. (2024, January 20–27). Omni-Kernel Network for Image Restoration. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
20. Wu, H., Qu, Y., Lin, S., Zhou, J., Qiao, R., Zhang, Z., Xie, Y., and Ma, L. (2021, January 20–25). Contrastive learning for compact single image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01041 ↗
21. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., and Shao, L. (2020, January 23–28). Learning enriched features for real image restoration and enhancement. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58595-2_30 ↗
22. Chen, D., He, M., Fan, Q., Liao, J., Zhang, L., Hou, D., Yuan, L., and Hua, G. (2019, January 7–11). Gated Context Aggregation Network for Image Dehazing and Deraining. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV.2019.00151 ↗
23. Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. (2022, January 18–24). MAXIM: Multi-axis MLP for image processing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00568 ↗
24. Zheng, Y., Zhan, J., He, S., Dong, J., and Du, Y. (2023, January 17–24). Curricular contrastive regularization for physics-aware single image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00560 ↗
25. Densely residual laplacian super-resolution10.1109/TPAMI.2020.3021088 ↗
26. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. (2018, January 18–23). Residual dense network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00262 ↗
27. Decomformer: Decompose Self-Attention of Transformer for Efficient Image Restoration10.1109/ACCESS.2024.3375360 ↗
28. Multi-stage image denoising with the wavelet transform10.1016/j.patcog.2022.109050 ↗
29. Li, C., Guo, C.L., Zhou, M., Liang, Z., Zhou, S., Feng, R., and Loy, C.C. (2023, January 1–5). Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.
30. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020, January 30). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
31. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 10–17). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
32. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
33. Zheng, C., Liu, X., Qi, G.J., and Chen, C. (2023, January 17–24). Potter: Pooling attention transformer for efficient human mesh recovery. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00161 ↗
34. The fast Fourier transform and its applications10.1109/TE.1969.4320436 ↗
35. Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., and Li, H. (2023, January 17–24). Internimage: Exploring large-scale vision foundation models with deformable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01385 ↗
36. Li, X., Wang, W., Hu, X., and Yang, J. (2019, January 15–20). Selective kernel networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00060 ↗
37. Abuolaim, A., and Brown, M.S. (2020, January 23–28). Defocus deblurring using dual-pixel data. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58607-2_7 ↗
38. Aifnet: All-in-focus image restoration network using a light field-based dataset10.1109/TCI.2021.3092891 ↗
39. Nah, S., Hyun Kim, T., and Mu Lee, K. (2017, January 21–26). Deep multi-scale convolutional neural network for dynamic scene deblurring. Proceedings of the IEEE/CVF International Conference on Computer Vision, Honolulu, HI, USA.10.1109/CVPR.2017.35 ↗
40. Shen, Z., Wang, W., Lu, X., Shen, J., Ling, H., Xu, T., and Shao, L. (November, January 27). Human-aware motion deblurring. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
41. Benchmarking single-image dehazing and beyond10.1109/TIP.2018.2867951 ↗
42. Ancuti, C.O., Ancuti, C., Sbert, M., and Timofte, R. (2019, January 22–25). Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. Proceedings of the IEEE International Conference on Image Processing, Taipei, Taiwan.10.1109/ICIP.2019.8803046 ↗
43. Ancuti, C.O., Ancuti, C., and Timofte, R. (2020, January 14–19). NH-HAZE: An image dehazing benchmark with non-homogeneous hazy and haze-free images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00230 ↗
44. Zhang, J., Cao, Y., Zha, Z.J., and Tao, D. (2020, January 12–16). Nighttime dehazing with a synthetic benchmark. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413763 ↗
45. Wei, C., Wang, W., Yang, W., and Liu, J. (2018, January 3–6). Deep retinex decomposition for low-light enhancement. Proceedings of the BMVC, Newcastle, UK.
46. Sparse gradient regularized deep retinex network for robust low-light image enhancement10.1109/TIP.2021.3050850 ↗
47. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
48. Loshchilov, I., and Hutter, F. (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv.
49. Lee, J., Son, H., Rim, J., Cho, S., and Lee, S. (2021, January 20–25). Iterative filter adaptive network for single image defocus deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00207 ↗
50. Image Restoration via Frequency Selection10.1109/TPAMI.2023.3330416 ↗
51. Dual-domain strip attention for image restoration10.1016/j.neunet.2023.12.003 ↗
52. Tsai, F.J., Peng, Y.T., Lin, Y.Y., Tsai, C.C., and Lin, C.W. (2022, January 23–27). Stripformer: Strip transformer for fast image deblurring. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19800-7_9 ↗
53. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., and Li, H. (2022, January 18–24). Uformer: A general u-shaped transformer for image restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01716 ↗
54. Liu, X., Ma, Y., Shi, Z., and Chen, J. (November, January 27). Griddehazenet: Attention-based multi-scale network for image dehazing. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
55. Restoring vision in hazy weather with hierarchical contrastive learning10.1016/j.patcog.2023.109956 ↗
56. Cai, Y., Bian, H., Lin, J., Wang, H., Timofte, R., and Zhang, Y. (2023, January 1–6). Retinexformer: One-stage retinex-based transformer for low-light image enhancement. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01149 ↗
57. Band representation-based semi-supervised low-light image enhancement: Bridging the gap between signal fidelity and perceptual quality10.1109/TIP.2021.3062184 ↗
58. Xu, X., Wang, R., Fu, C.W., and Jia, J. (2022, January 18–24). SNR-aware low-light image enhancement. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01719 ↗