Virtual restoration method of Kizil Grotto murals based on multimodal controlled diffusion models
Source abstract
Abstract
Generative deep learning provides new approaches for natural image restoration and structural reconstruction. However, virtually restoring Kizil cave murals remains difficult due to their unique artistic style, complex damage, and the need to preserve semantic consistency. This study proposes a multimodal controlled diffusion model that integrates textual and multi-dimensional visual features for high-precision restoration. The model leverages latent space diffusion for high-quality image generation and introduces structural constraints to improve semantic alignment and controllability. A dynamic feature-adaptive GSC (DFA-GSC) module captures local and global features through multi-scale convolution and an adaptive weight generator, enhancing texture perception. For damaged regions, a conditional matching loss helps refine both texture and structure. Experimental results demonstrate that compared to traditional CNNs and single diffusion models, the proposed method achieves superior performance in both evaluation metrics and visual quality.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Aesthetic Perception and Analysis · Computer Graphics and Visualization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (49)
Show source-provided references
1. Ma, X. L. Zhao Li Committee Member: Let Digital Empowerment Bring the Kizil Grottoes to Life (Asia Center Times, 2025).
2. Chen, H. et al. DualAST: dual style-learning networks for artistic style transfer. In Proc the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition 872–881 (IEEE, 2021).10.1109/CVPR46437.2021.00093 ↗
3. Haliassos, A., Barmpoutis, P., Stathaki, T., Quirke, S., Constantinides, A. Classification and detection of symbols in ancient papyri. In Visual Computing for Cultural Heritage (eds, Liarokapis, F., Voulodimos, A., Doulamis, N. & Doulamis, A.) 121–140 (Springer Series on Cultural Computing, 2020).10.1007/978-3-030-37191-3_7 ↗
4. Bertalmio, M., Sapiro, G., Caselles, V. & Ballester, C. Image inpainting. In Proc the 27th ACM SIGGRAPH Conference on Computer Graphics and Interactive Techniques 417–424 (ACM Digital, 2000).10.1145/344779.344972 ↗
5. Efros, A. A. & Freeman, W. T. Image quilting for texture synthesis and transfer. In Proc the 28th ACM SIGGRAPH Conference on Computer Graphics and Interactive Techniques 341–346 (ACM Digital Library, 2001).10.1145/383259.383296 ↗
6. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G. & Verdera, J. Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10, 1200–1211 (2001).10.1109/83.935036 ↗
7. Chen, Y., Ai, Y. & Guo, H. Improved curvature driven model for Dunhuang fresco restoration algorithm. J. Comput. Aided Des. Comput. Graph. 32, 787–796 (2020).
8. Criminisi, A., Perez, P. & Toyama, K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 1200–1212 (2004).10.1109/TIP.2004.833105 ↗
9. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B. & Sen, P. Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, Article 82 (2012).10.1145/2185520.2185578 ↗
10. Hays, J. & Efros, A. A. Scene completion using millions of photographs. Commun. ACM 51, 87–94 (2008).10.1145/1400181.1400202 ↗
11. Barnes, C., Shechtman, E., Finkelstein, A. & Goldman, D. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 24–34 (2009).10.1145/1531326.1531330 ↗
12. Shen, J., Kang, S. H. & Chan, T. F. Euler’s elastica and curvature-based inpainting. SIAM J. Appl. Math. 63, 564–592 (2003).10.1137/S0036139901390088 ↗
13. Lu, X. B. & Wang, W. L. Improvement of damaged Thangka image repair algorithm based on sample block. J. Comput. Appl. 30, 943–946 (2010).
14. Yao, F. Damaged region filling by improved Criminisi image inpainting algorithm for Thangka. Clust. Comput. 22, 13683–13691 (2019).10.1007/s10586-018-2068-4 ↗
15. Zeng, Y., Fu, J., Chao, H. & Guo, B. Learning pyramid-context encoder network for high-quality image inpainting. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition 1486–1494 (IEEE, 2019).10.1109/CVPR.2019.00158 ↗
16. Li, J., He, F., Zhang, L., Du, B. & Tao, D. Progressive reconstruction of visual structure for image inpainting. In 2019 IEEE/CVF International Conference on Computer Vision 5961–5970 (IEEE, 2019).10.1109/ICCV.2019.00606 ↗
17. Suvorov, R. et al. Resolution-robust large mask inpainting with Fourier convolutions. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision 3172–3182 (IEEE, 2021).10.1109/WACV51458.2022.00323 ↗
18. Zheng, C. et al. Bridging global context interactions for high-fidelity pluralistic image completion. IEEE Trans. Pattern Anal. Mach. Intell. 46, 8320–8333 (2024).10.1109/TPAMI.2024.3403695 ↗
19. Nazeri, K., Ng, E., Joseph, T., Qureshi, F. Z. & Ebrahimi, M. EdgeConnect: structure guided image inpainting using edge prediction. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) 3265–3274 (IEEE, 2019).10.1109/ICCVW.2019.00408 ↗
20. Zhao, Y., Price, B., Cohen, S. & Gurari, D. Guided image inpainting: replacing an image region by pulling content from another image. In: 2019 IEEE Winter Conference on Applications of Computer Vision 1514–1523 (IEEE, 2019).10.1109/WACV.2019.00166 ↗
21. Zhang, L., Chen, Q., Hu, B. & Jiang, S. Text-guided neural image inpainting. In Proc of the 28th ACM International Conference on Multimedia 1302–1310 (IEEE, 2020).10.1145/3394171.3414017 ↗
22. Guo, X., Yang, H. & Huang, D. Image inpainting via conditional texture and structure dual generation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 14114–14123 (IEEE, 2021).10.1109/ICCV48922.2021.01387 ↗
23. Wan, Z., Zhang, J., Chen, D. & Liao, J. High-fidelity pluralistic image completion with transformers. In 2021 IEEE/CVF International Conference on Computer Vision 4672–4681 (IEEE, 2021).10.1109/ICCV48922.2021.00465 ↗
24. Xu, W. & Fu, Y. Deep learning algorithm in ancient relics image colour restoration technology. Multimed. Tools Appl. 82, 23119–23150 (2023).10.1007/s11042-022-14108-z ↗
25. Peng, X. et al. C3N: content-constrained convolutional network for mural image completion. Neural Comput. Appl. 35, 1959–1970 (2023).10.1007/s00521-022-07806-0 ↗
26. Yang, J., Ruhaiyem, N. I. & Zhou, C. A 3M-hybrid model for the restoration of unique giant murals: a case study on the murals of Yongle Palace. IEEE Access 13, 38809–38824 (2025).10.1109/ACCESS.2025.3542320 ↗
27. Wang, F., Wu, K. & Cao, Y. High resolution mural images inpainting based on dual aggregation generative adversarial network. In 2023 International Conference on Image Processing, Computer Vision and Machine Learning 97–102 (IEEE, 2023).10.1109/ICICML60161.2023.10424787 ↗
28. Wang, J. et al. Image editing with diffusion models: a survey. https://arxiv.org/abs/2504.13226 (2025).
29. Lugmayr, A. et al. RePaint: inpainting using denoising diffusion probabilistic models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition 11451–11461 (IEEE, 2022).10.1109/CVPR52688.2022.01117 ↗
30. Meng, C. et al. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In The Tenth International Conference on Learning Representations (ICLR, 2022).
31. Wang, Y., Yu, J. & Zhang, J. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations (ICLR, 2023).
32. Xia, B. et al. (2023). DiffIR: efficient diffusion model for image restoration. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) 13049–13059 (IEEE, 2023).10.1109/ICCV51070.2023.01204 ↗
33. Zhang, L. et al. Minutes to seconds: speeded-up DDPM-based image inpainting with coarse-to-fine sampling. In 2024 IEEE International Conference on Multimedia and Expo (ICME) 1–6 (IEEE, 2024).10.1109/ICME57554.2024.10687818 ↗
34. Saharia, C. et al. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, Article 15 (ACM, 2022).10.1145/3528233.3530757 ↗
35. Huang, Y. et al. WaveDM: wavelet-based diffusion models for image restoration. IEEE Trans. Multimed. 26, 7058–7073 (2023).10.1109/TMM.2024.3359769 ↗
36. Huang, J. et al. Dual-schedule inversion: training- and tuning-free inversion for real image editing. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision 660–669 (IEEE, 2024).10.1109/WACV61041.2025.00074 ↗
37. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML) 8748–8763 (PMLR, 2021).
38. Zhang, L., Rao, A. & Agrawala, M. Adding conditional control to text-to-image diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) 3813–3824 (IEEE, 2023).10.1109/ICCV51070.2023.00355 ↗
39. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. https://arxiv.org/abs/1312.6114 (2013).
40. Rombach, R., Blattmann, A., Lorenz, D., Esser, P. & Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 10684–10695 (IEEE, 2022).10.1109/CVPR52688.2022.01042 ↗
41. Woo, S., Park, J., Lee, J.-Y. & Kweon, I. S. CBAM: Convolutional Block Attention Module. In Computer Vision – ECCV 2018, Lecture Notes in Computer Science, vol. 11211, 3–19 (Springer, Cham, 2018).10.1007/978-3-030-01234-2_1 ↗
42. Ho, J., Jain, A. & Abbeel, P. Denoising Diffusion Probabilistic Models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 574–585 (Curran Associates Inc., Red Hook, NY, 2020).
43. Li, M. et al. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback. In European Conference on Computer Vision (ECCV, 2024).10.1007/978-3-031-72667-5_8 ↗
44. Yang, S., Jiang, L., Liu, Z. & Loy, C. C. Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7683–7692 (IEEE, 2022).10.1109/CVPR52688.2022.00754 ↗
45. Ju, X. et al. BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion. In Computer Vision – ECCV 2024, 150–168 (2024).10.1007/978-3-031-72661-3_9 ↗
46. Li, L. et al. Line Drawing Guided Progressive Inpainting for Mural Damage. J. Comput. Cult. Herit. (2025).10.1145/3737703 ↗
47. Gupta, P., Srivastava, P., Bhardwaj, S. & Bhateja, V. A modified PSNR metric based on HVS for quality assessment of color images. In Proc the 2011 International Conference on Communication and Industrial Application 1–4 (ICCIA, 2011).10.1109/ICCIndA.2011.6146669 ↗
48. Horé, A. & Ziou, D. Image quality metrics: PSNR vs. SSIM. In Proc the 2010 20th International Conference on Pattern Recognition 2366–2369 (ICPR, 2010).10.1109/ICPR.2010.579 ↗
49. Zhang, R., Isola, P., Efros, A. A., Shechtman, E. & Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 586–595 (IEEE, 2018).10.1109/CVPR.2018.00068 ↗