Why Not Both? An Attention-Guided Transformer with Pixel-Related Deconvolution Network for Face Super-Resolution
Source abstract
Abstract
Transformer-based encoder-decoder networks for face super-resolution (FSR) have achieved promising success in delivering stunningly clear and detailed facial images by capturing local and global dependencies. However, these methods have certain limitations. Specifically, the deconvolution in upsampling layers neglects the relationship between adjacent pixels, which is crucial in facial structure reconstruction. Additionally, raw feature maps are fed to the transformer blocks directly without mining their potential feature information, resulting in suboptimal face images. To circumvent these problems, we propose an attention-guided transformer with pixel-related deconvolution network for FSR. Firstly, we devise a novel Attention-Guided Transformer Module (AGTM), which is composed of an Attention-Guiding Block (AGB) and a Channel-wise Multi-head Transformer Block (CMTB). AGTM at the top of the encoder-decoder network (AGTM-T) promotes both local facial details and global facial structures, while AGTM at the bottleneck side (AGTM-B) optimizes the encoded features. Secondly, a Pixel-Related Deconvolution (PRD) layer is specially designed to establish direct relationships among adjacent pixels in the upsampling process. Lastly, we develop a Multi-scale Feature Fusion Module (MFFM) to fuse multi-scale features for better network flexibility and reconstruction results. Quantitative and qualitative experimental results on various datasets demonstrate that the proposed method outperforms other state-of-the-art FSR methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Image and Signal Denoising Methods · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China
Funding
National Natural Science Foundation of China
References (64)
Show source-provided references
1. Baker, S., and Kanade, T. (2000, January 28–30). Hallucinating faces. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France.
2. Deep learning-based face super-resolution: A survey
3. An edge-guided image interpolation algorithm via directional filtering and data fusion10.1109/TIP.2006.877407 ↗
4. Super-resolution of face images using kernel pca-based prior10.1109/TMM.2007.893346 ↗
5. Position-patch based face hallucination using convex optimization10.1109/LSP.2011.2140370 ↗
6. Tappen, M.F., and Liu, C. (2012, January 7–13). A bayesian approach to alignment-based image hallucination. Proceedings of the European Conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33786-4_18 ↗
7. Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression10.1109/TIP.2012.2208977 ↗
8. Face Super-Resolution via Multilayer Locality-Constrained Iterative Neighbor Embedding and Intermediate Dictionary Learning10.1109/TIP.2014.2347201 ↗
9. Zhang, K., Zhang, Z., Cheng, C.W., Hsu, W.H., Qiao, Y., Liu, W., and Zhang, T. (2018, January 8–14). Super-identity convolutional neural network for face hallucination. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01252-6_12 ↗
10. Learning spatial attention for face super-resolution10.1109/TIP.2020.3043093 ↗
11. Lu, T., Wang, Y., Zhang, Y., Wang, Y., Wei, L., Wang, Z., and Jiang, J. (2021, January 20–24). Face hallucination via split-attention in split-attention network. Proceedings of the ACM International Conference on Multimedia, Chengdu, China.10.1145/3474085.3475682 ↗
12. Yang, T., Ren, P., Xie, X., and Zhang, L. (2021, January 20–25). Gan prior embedded network for blind face restoration in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00073 ↗
13. Yang, D., Wei, Y., Hu, C., Yu, X., Sun, C., Wu, S., and Zhang, J. (2023). Multi-Scale Feature Fusion and Structure-Preserving Network for Face Super-Resolution. Appl. Sci., 13.10.3390/app13158928 ↗
14. Wang, Z., Zhang, J., Chen, R., Wang, W., and Luo, P. (2022, January 18–24). Restoreformer: High-quality blind face restoration from undegraded key-value pairs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01699 ↗
15. Wang, Z.D., Cun, X.D., Bao, J.M., Zhou, W.G., Liu, J.Z., and Li, H.Q. (2022, January 18–24). Uformer: A General U-Shaped Transformer for Image Restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01716 ↗
16. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., and Tan, M. (2020, January 14–19). Closed-loop matters: Dual regression networks for single image superresolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR42600.2020.00545 ↗
17. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. (2021, January 19–25). SwinIR: Image Restoration Using Swin Transformer. Proceedings of the IEEE International Conference on Computer Vision Workshops, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00210 ↗
18. CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution10.1109/TIP.2023.3261747 ↗
19. Wang, C., Jiang, J., Zhong, Z., and Liu, X. (2023, January 18–24). Spatial-Frequency Mutual Learning for Face Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02141 ↗
20. Lai, W.S., Huang, J.B., Ahuja, N., and Yang, M.H. (2017, January 22–25). Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.618 ↗
21. Leland, M., John, H., and James, M. (2020). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv.
22. Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep learning face attributes in the wild. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.425 ↗
23. Le, V., Brandt, J., Lin, Z., Bourdev, L., and Huang, T.S. (2012, January 7–13). Interactive facial feature localization. Proceedings of the European Conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33712-3_49 ↗
24. Deep learning for image super-resolution: A survey10.1109/TPAMI.2020.2982166 ↗
25. Li, J., Pei, Z., and Zeng, T. (2021). From beginner to master: A survey for deep learning-based single-image super-resolution. arXiv.
26. Zhou, E.J., Fan, H.Q., Cao, Z.M., Jiang, Y.N., and Yin, Q. (2015, January 25–30). Learning face hallucination in the wild. Proceedings of the Association for the Advancement of Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9795 ↗
27. Cao, Q.X., Lin, L., Shi, Y.K., Liang, X.D., and Li, G.B. (2017, January 22–25). Attention-aware face hallucination via deep reinforcement learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.180 ↗
28. Wavelet domain generative adversarial network for multiscale face hallucination10.1007/s11263-019-01154-8 ↗
29. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z.H. (2017, January 22–25). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
30. Yang, L.B., Liu, C., Wang, P., Wang, S.S., Ren, P.R., Ma, S.W., and Gao, W. (2020, January 8–11). Hifacegan: Face renovation via collaborative suppression and replenishment. Proceedings of the ACM International Conference on Multimedia, Dublin, Ireland.10.1145/3394171.3413965 ↗
31. Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes
32. Dou, H., Chen, C., Hu, X.Y., Xuan, Z.X., Hu, Z.S., and Peng, S.L. (2020, January 8–11). Pca-srgan: Incremental orthogonal projection discrimination for face super-resolution. Proceedings of the ACM International Conference on Multimedia, Dublin, Ireland.10.1145/3394171.3413590 ↗
33. Supervised pixel-wise GAN for face super-resolution10.1109/TMM.2020.3006414 ↗
34. Chen, Y., Tai, Y., Liu, X., Shen, C., and Yang, J. (2018, January 18–23). Fsrnet: End-to-end learning face super-resolution with facial priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00264 ↗
35. Bulat, A., and Tzimiropoulos, G. (2018, January 18–23). Super-fan: Integrated facial landmark localization and super-resolution of real-world low-resolution faces in arbitrary poses with GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00019 ↗
36. Hu, X., Ren, W., LaMaster, J., Cao, X., Li, X., Li, Z., Menze, B., and Liu, W. (2020, January 23–28). Face super-resolution guided by 3d facial priors. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58548-8_44 ↗
37. Ma, C., Jiang, Z., Rao, Y., Lu, J., and Zhou, J. (2020, January 14–19). Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00561 ↗
38. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
39. Li, Z.Z., Li, G., Li, T., Liu, S., and Gao, W. (2021, January 20–24). Information-Growth Attention Network for Image Super-Resolution. Proceedings of the ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475207 ↗
40. Kalarot, R., Li, T., and Porikli, F. (2020, January 1–5). Component Attention Guided Face Super-Resolution Network: CAGFace. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, CO, USA.10.1109/WACV45572.2020.9093399 ↗
41. Li, C., and Xiao, N. (2022, January 21–25). A Face Structure Attention Network for Face Super-Resolution. Proceedings of the International Conference on Pattern Recognition (ICPR), Montreal, QC, Canada.10.1109/ICPR56361.2022.9956640 ↗
42. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Virtual.
43. Xiong, L., Zhang, J., Zheng, X., and Wang, Y. (2024). Context Transformer and Adaptive Method with Visual Transformer for Robust Facial Expression Recognition. Appl. Sci., 14.10.3390/app14041535 ↗
44. Shi, A., and Ding, H. (2023). Underwater Image Super-Resolution via Dual-aware Integrated Network. Appl. Sci., 13.10.3390/app132412985 ↗
45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
46. Nonlinear dimensionality reduction by locally linear embedding10.1126/science.290.5500.2323 ↗
47. Zhang, Z., Qi, C., and Asif, M.R. (2018, January 12–16). Investigation on Projection Space Pairs in Neighbor Embedding Algorithms. Proceedings of the IEEE International Conference on Signal Processing, Beijing, China.10.1109/ICSP.2018.8652441 ↗
48. Face Hallucination Based on Modified Neighbor Embedding and Global Smoothness Constraint10.1109/LSP.2014.2329473 ↗
49. Tu, Q., Li, J.W., and Javaria, I. (2016, January 25–28). Locality constraint neighbor embedding via KPCA and optimized reference patch for face hallucination. Proceedings of the IEEE International Conference on Image Processing, Phoenix, AZ, USA.10.1109/ICIP.2016.7532392 ↗
50. Reference-Guided Deep Super-Resolution via Manifold Localized External Compensation10.1109/TCSVT.2018.2838453 ↗
51. Menon, S., Damian, A., Hu, S., Ravi, N., and Rudin, C. (2020, January 14–19). PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00251 ↗
52. Multi-Stage Degradation Homogenization for Super-Resolution of Face Images With Extreme Degradations10.1109/TIP.2021.3086595 ↗
53. Faucher, C., Hassell, J., and Potter, M. (2020). Deep Learning for Coders with fastai and PyTorch, O’Reilly Media.
54. Newell, A., Yang, K., and Deng, J. (2016, January 11–14). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
55. Li, X., Wang, W., Hu, X., and Yang, J. (2019, January 16–20). Selective Kernel Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00060 ↗
56. Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv.
57. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M. (2022, January 18–24). Restormer: Efficient Transformer for High-Resolution Image Restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00564 ↗
58. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
59. Image information and visual quality10.1109/TIP.2005.859378 ↗
60. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
61. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z.M., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4–9). Automatic differentiation in pytorch. Proceedings of the Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA.
62. (2024, April 20). NumPy. Available online: https://numpy.org/.
63. Huang, G.B., Mattar, M., Berg, T., and Learned-Miller, E. (2007). Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments, University of Massachusetts. Technical Report 07-49.
64. Serengil, S.I., and Ozpinar, A. (2020, January 15–17). Lightface: A hybrid deep face recognition framework. Proceedings of the Innovations in Intelligent Systems and Applications Conference, Istanbul, Turkey.10.1109/ASYU50717.2020.9259802 ↗