Mural inpainting via two-stage generative adversarial network
Source abstract
Abstract
The digital restoration of Dunhuang murals is of extremely high value for the research and dissemination of mural culture and art. However, a large number of murals have been damaged to varying degrees. In this paper, we propose a two-stage coarse-to-fine digital mural restoration framework to solve the large area of irregular shape damage on the mural. The first stage is used to achieve coarse-grained semantic reconstruction, and the second stage is used to achieve fine-grained feature reconstruction. To improve the repair quality, we also designed a new building block (STMA) that integrates the Swin transformer module (SwinT) and the multi-scale dilated convolution attention module (MSDA). Meanwhile, the proposed loss function is to further empower the proposed model to repair damaged murals. Extensive comparative experiments show that the proposed model can effectively restore the missing content of the mural and exceed the comparative methods in both quantitative and qualitative evaluation.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (52)
Show source-provided references
1. Xu, W. & Fu, Y. Deep learning algorithm in ancient relics image colour restoration technology. Multimedia Tools Appl 82, 23119–23150 (2023).10.1007/s11042-022-14108-z ↗
2. Liu, W. et al. Multi-stage progressive reasoning for dunhuang murals inpainting. In Proc. 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML). 211–217 (IEEE, 2023).10.1109/PRML59573.2023.10348363 ↗
3. Zheng, C., Cham, T. J., Cai, J. & Phung, D. Bridging Global Context Interactions for High-Fidelity Image Completion. In Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11502–11512 (IEEE, 2022).10.1109/CVPR52688.2022.01122 ↗
4. Zeng, Y., Fu, J., Chao, H. & Guo, B. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transac. Visualiz. Comput. Gr. 29, 3266–3280 (2023).10.1109/TVCG.2022.3156949 ↗
5. Quan, W. et al. Image Inpainting with Local and Global Refinement. IEEE Transac. Image Process. 31, 2405–2420 (2022).10.1109/TIP.2022.3152624 ↗
6. Li, Y. et al. Efficient and explicit modelling of image hierarchies for image restoration. In Proc. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada. 18278–18289 (IEEE, 2023).10.1109/CVPR52729.2023.01753 ↗
7. Yang, J., Ruhaiyem, N. I. R. & Zhou, C. A 3M-Hybrid model for the restoration of unique giant murals: a case study on the murals of Yongle Palace. IEEE Access 13, 38809–38824 (2025).10.1109/ACCESS.2025.3542320 ↗
8. Liang, J., Cao, J., Sun, G., Zhang, K. & Timofte, R. SwinIR: image restoration using swin transformer. In Proc. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Montreal, BC, Canada, 1833–1844, (IEEE, 2021).10.1109/ICCVW54120.2021.00210 ↗
9. Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada, 9992–10002 (IEEE, 2021).10.1109/ICCV48922.2021.00986 ↗
10. Chen, G., Gao, Z., Zhu, P. & Chen, Z. Learning a multi-scale deep residual network of dilated-convolution for image denoising. In Proc. 2020 IEEE 5th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), Chengdu, China, 348–353 (IEEE, 2020).10.1109/ICCCBDA49378.2020.9095754 ↗
11. Ople, J. J. M., Yeh, P. Y., Sun, S. W., Tsai, I. T. & Hua, K. L. Multi-scale neural network with dilated convolutions for image deblurring. IEEE Access 8, 53942–53952 (2020).10.1109/ACCESS.2020.2980996 ↗
12. Burger, H. C., Schuler, C. & Harmeling, S. Learning how to combine internal and external denoising methods. In Proc. Pattern Recognition: 35th German Conference, GCPR 2013, Saarbrücken, Germany, 8142, 121–130 (2013).10.1007/978-3-642-40602-7_13 ↗
13. Zhang, K. et al. Practical blind image denoising via Swin-Conv-UNet and data synthesis. Mach. Intell. Res. 20, 822–836 (2023).10.1007/s11633-023-1466-0 ↗
14. Goodfellow, I. et al. Generative adversarial networks. Commun ACM 63, 139–144 (2020).10.1145/3422622 ↗
15. Liu, H. et al. PD-GAN: Probabilistic Diverse GAN for Image Inpainting. In Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA, 9367–9376 (IEEE, 2021).10.1109/CVPR46437.2021.00925 ↗
16. Nazeri, K., Ng, E., Joseph, T. Qureshi, F. & Ebrahimi, M. EdgeConnect: Structure-Guided image inpainting using edge prediction. In Proc. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South), 3265–3274 (IEEE, 2019).10.1109/ICCVW.2019.00408 ↗
17. Wan, Z., Zhang, J., Chen, D. & Liao, J. High-fidelity pluralistic image completion with transformers. In Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada, 4672–4681 (IEEE, 2021).10.1109/ICCV48922.2021.00465 ↗
18. Jiang, J. et al. Research on digital image restoration technology of Xizang Murals Based on CDD Model. Electron. Design Eng. 22, 177–179 (2014).
19. Chen, Y., Ai, Y. & Guo, H. Inpainting Algorithm for Dunhuang mural based on improved curvature-driven diffusion model. J. Comput. Aided Design Comput. Gr. 32, 787–796 (2020).
20. Wu, M. & Wang, H. Research on multi-scale detection and image inpainting of Tang dynasty tomb murals. Comput. Eng. Appl. 52, 169–174 (2016).
21. Cao, J., Li, Y., Zhang, Q. & Cui, H. Restoration of an ancient temple mural by a local search algorithm of an adaptive sample block. Heritage Sci. 7, 39 (2019).10.1186/s40494-019-0281-y ↗
22. Yang, X. P. & Wang, S. W. Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm. J. Comput. Aided Design Comput. Gr. 23, 284–289 (2011).
23. Chen, Y., Chen, J. & Tao, M. F. Mural inpainting progressive generative adversarial networks based on structure guided. J. Beijing Univ. Aeronaut. Astronaut. 49, 1247–1259 (2023).
24. Chen, Y., Zhao, M. X. & Tao, M. F. Mural inpainting algorithm for group sparse based on multi-scale contourlet transform decomposition. J. Xidian Univ. 49, 120–128 (2022).
25. Chen, Y., Ai, Y. P. & Chen, J. Algorithm for Dunhuang Mural inpainting based on Gabor transform and group sparse representation. Laser Optoelectron. Progress 57, 175–184 (2020).10.3788/LOP57.111400 ↗
26. Cao, J., Zhang, Z., Zhao, A., Cui, H. & Zhang, Q. Ancient mural restoration based on a modified generative adversarial network. Heritage Sci. 8, 7 (2020).10.1186/s40494-020-0355-x ↗
27. Li, L. et al. Line Drawing Guided Progressive Inpainting of Mural Damages. arXiv preprint arXiv: 2211.06649 (2022).
28. Chen, Y., Chen, J. & Tao, M. F. Mural inpainting with generative adversarial networks based on multi-scale feature and attention fusion. J. Beijing Univ. Aeronaut. Astronaut. 49, 254–264 (2023).
29. Li, J., Shi, Y., Liu, W., Wang, J. & Du, S. Progressive dunhuang murals inpainting based on attention mechanism. In Proc. 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML). Urumqi, China, 350–356 (IEEE, 2023).10.1109/PRML59573.2023.10348292 ↗
30. Wang, N., Wang, W., Hu, W., Fenster, A. & Li, S. Thanka mural inpainting based on multi-scale adaptive partial convolution and stroke-like mask. IEEE Transac. Image Process. 30, 3720–3733 (2021).10.1109/TIP.2021.3064268 ↗
31. Deng, X. & Yu, Y. Ancient mural inpainting via structure information guided two-branch model. Heritage Sci. 11, 131 (2023).10.1186/s40494-023-00972-x ↗
32. Khaled, A. BCN: Batch Channel Normalization for Image Classification. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Springer, Cham,15311, 295–308 (2025).10.1007/978-3-031-78195-7_20 ↗
33. Oyedotun, O. K., Aouada, D. & Ottersten, B. Going deeper with neural networks without skip connections. In Proc. 2020 IEEE International Conference on Image Processing (ICIP). Abu Dhabi, United Arab Emirates, 1756–1760 (IEEE, 2020).10.1109/ICIP40778.2020.9191356 ↗
34. Mao, X. J., Shen, C. & Yang, Y. B. Image restoration using convolutional auto-encoders with symmetric skip connections. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16). Barcelona, Spain, 2810–2818 (2016).
35. Miyato, T., Kataoka, T., Koyama, M. & Yoshida, Y. Spectral Normalization for Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR, 2018).
36. Qi, G. J. Loss-sensitive generative adversarial networks on lipschitz densities. Int. J. Comput. Vision 128, 1118–1140 (2020).10.1007/s11263-019-01265-2 ↗
37. Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, 5987–5995 (IEEE, 2017).10.1109/CVPR.2017.634 ↗
38. Tolstikhin, I. O. et al. MLP-Mixer: An all-MLP Architecture for Vision. Adv. Neural Inform. Process. Syst. 34, 24264–24272 (2021).
39. Xiang, X., Liu, M., Zhang, S., Wei, P. & Chen, B. Multi-scale attention and dilation network for small defect detection. Pattern Recognit. Lett. 172, 82–88 (2023).10.1016/j.patrec.2023.06.010 ↗
40. Yu, F. & Koltun, V. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations (ICLR). 1–9 (ICLR, 2016).
41. Zhang, Y. et al. Image super-resolution using very deep residual channel attention networks. In Proc. Eur Conf Comput Vis (ECCV). 11211, 294–310 (2018).
42. Hou, Q., Zhou, D. & Feng, J. Coordinate Attention for Efficient Mobile Network Design. In Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA, 13708–13717 (IEEE, 2021).10.1109/CVPR46437.2021.01350 ↗
43. Zhao, H., Gallo, O., Frosio, I. & Kautz, J. Loss functions for image restoration with neural networks. IEEE Transact. Comput. Imaging 3, 47–57 (2017).10.1109/TCI.2016.2644865 ↗
44. Risser, E., Wilmot, P. & Barnes, C. Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893 (2017).
45. Liu, G., Reda, F. A., Shih, K. J., Wang, T. C., Tao, A. & Catanzaro, B. Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds) Computer Vision – ECCV 2018. 11215, 89–105 (2018).10.1007/978-3-030-01252-6_6 ↗
46. Poma, X. S., Riba, E. & Sappa, A. Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection. In Proc. IEEE/CVF winter conference on applications of computer vision (WACV). Snowmass, CO, USA, 1912–1921 (IEEE, 2020).10.1109/WACV45572.2020.9093290 ↗
47. Guo, X., Yang, H. & Huang, D. Image Inpainting via Conditional Texture and Structure Dual Generation. In Proc. IEEE/CVF international conference on computer vision (ICCV), Montreal, QC, Canada, 14134–14143 (IEEE, 2021).10.1109/ICCV48922.2021.01387 ↗
48. Li, W. et al. MAT: Mask-aware transformer for large hole image inpainting. In Proc. IEEE/CVF conference on computer vision and pattern recognition (CVPR). New Orleans, LA, USA, 10758–10768 (IEEE, 2022).10.1109/CVPR52688.2022.01049 ↗
49. Horé, A. & Ziou, D. Image Quality Metrics: PSNR vs. SSIM. In Proc. 20th International Conference on Pattern Recognition, 2366–2369 (IEEE, 2010).10.1109/ICPR.2010.579 ↗
50. Allen, D. M. Mean square error of prediction as a criterion for selecting variables. Technometrics 13, 469–475 (1971).10.1080/00401706.1971.10488811 ↗
51. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17) 30, 6629–6640 (2017).
52. Zhang, R., Isola, P., Efros, A. A., Shechtman, E. & Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE conference on computer vision and pattern recognition(CVPR). Salt Lake City, UT, USA, 586–595 (IEEE, 2018).10.1109/CVPR.2018.00068 ↗