Progressive enhancement and restoration for mural images under low-light and defective conditions based on multi-receptive field strategy
Source abstract
Abstract
Ancient murals represent invaluable heritage, providing deep insights into historic culture. However, these murals are increasingly at risk due to long-term degradation caused by oxidation and inadequate protection, and other factors, resulting in damages such as peeling and mold. Furthermore, the challenge posed by low-light conditions during image capture exacerbates the analyses and the restoration process, making it difficult to effectively identify and repair defects. To tackle these pressing challenges and facilitate efficient batch restoration at archeological sites, we propose a two-stage restoration model named MER. First, our model employs an innovative illumination enhancement module to improve the lighting of low-light mural images. Second, an automatic defect detection strategy, combined with a multi-receptive field approach, is utilized to systematically restore the identified defects. Comprehensive evaluations demonstrate that our MER model significantly enhances the visual quality of the restored images and achieves superior performance on relevant metrics compared to existing methods. Our works highlight the importance of addressing both lighting issues and defect detection in ancient mural restoration. Furthermore, we have launched a website dedicated to the restoration of ancient mural paintings, utilizing the proposed model. Code is available at https://gitee.com/bbfan2024/MER.git .
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Image Enhancement Techniques · Advanced Image Processing Techniques · Image and Signal Denoising Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (47)
Show source-provided references
1. Song, G. & Wang, H. Artificial intelligence-assisted Fresco restoration with multiscale line drawing generation. Complexity 4, 1–12 (2021).
2. Izzo, F. C., Falchi, L., Zendri, E. & Zendri, E. A Study on Materials and Painting Techniques of 1930s Italian Mural Paintings: Two Cases by Mario Sironi and Edmondo Bacci in Venice (Cambridge Scholar Publishing, 2015).
3. Wang, H.-L. et al. Dunhuang mural restoration using deep learning. In Proc. SIGGRAPH Asia 2018 Technical Briefs, 1–4 (2018).10.1145/3283254.3283263 ↗
4. Yong, C., Jin, C. & Meifeng, T. Mural inpainting generative adversarial networks based on multi-scale feature and attention fusion. J. Beijing Univ. Aeronaut. Astronaut. 49, 254–264 (2021).10.2514/1.J050826 ↗
5. Arici, T., Dikbas, S. & Altunbasak, Y. A histogram modification framework and its application for image contrast enhancement. IEEE Trans. Image Process. 18, 1921–1935 (2009).10.1109/TIP.2009.2021548 ↗
6. Lore, K. G., Akintayo, A. & Sarkar, S. LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recognit. 61, 650–662 (2017).10.1016/j.patcog.2016.06.008 ↗
7. Zhang, Y., Zhang, J. & Guo, X. Kindling the darkness: a practical low-light image enhancer. In Proc. 27th ACM International Conference on Multimedia, 1632–1640 (2019).10.1145/3343031.3350926 ↗
8. Wang, R. et al. Underexposed photo enhancement using deep illumination estimation. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6842–6850 (2019).10.1109/CVPR.2019.00701 ↗
9. Yang, W., Wang, S., Fang, Y., Wang, Y. & Liu, J. From fidelity to perceptual quality: a semi-supervised approach for low-light image enhancement. In Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3060–3069 (2020).10.1109/CVPR42600.2020.00313 ↗
10. Zhang, Y., Di, X., Zhang, B. & Wang, C. Self-supervised image enhancement network: training with low light images only. Preprint at https://arxiv.org/abs/2002.11300 (2020).
11. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G. & Verdera, J. Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10, 1200–1211 (2001).10.1109/83.935036 ↗
12. Tschumperlé, D. & Deriche, R. Vector-valued image regularization with PDEs: a common framework for different applications. IEEE Trans. Pattern Anal. Mach. Intell. 27, 506–517 (2005).10.1109/TPAMI.2005.87 ↗
13. Bertalmio, M., Sapiro, G., Caselles, V. & Ballester, C. Image inpainting. In Proc. ACM SIGGRAPH, 417–424 (2000).10.1145/344779.344972 ↗
14. Levin, A., Zomet, A. & Weiss, Y. Learning how to inpaint from global image statistics. In Proc. IEEE International Conference on Computer Vision (ICCV) (2003).10.1109/ICCV.2003.1238360 ↗
15. Huang, J.-B., Kang, S. B., Ahuja, N. & Kopf, J. Image completion using planar structure guidance. ACM Trans. Graph. 33, 129:1–129:10 (2014).10.1145/2601097.2601205 ↗
16. Kwatra, V., Essa, I., Bobick, A. & Kwatra, N. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 795–802 (2005).10.1145/1073204.1073263 ↗
17. Simakov, D., Caspi, Y., Shechtman, E. & Irani, M. Summarizing visual data using bidirectional similarity. In Proc. 2008 IEEE Conference on Computer Vision and Pattern Recognition, 1–8 (2008).10.1109/CVPR.2008.4587842 ↗
18. Criminisi, A., Perez, P. & Toyama, K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 1200–1212 (2004).10.1109/TIP.2004.833105 ↗
19. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B. & Sen, P. Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. https://doi.org/10.1145/2185520.2185578 (2012).10.1145/2185520.2185578 ↗
20. Efros, A. A. & Leung, T. K. Texture synthesis by nonparametric sampling. In Proc. Seventh IEEE International Conference on Computer Vision, 1033–1038 (1999).10.1109/ICCV.1999.790383 ↗
21. Barnes, C., Shechtman, E., Finkelstein, A. & Goldman, D. B. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 24:1–24:11 (2009).10.1145/1531326.1531330 ↗
22. Yu, J. et al. Generative image inpainting with contextual attention. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5505–5514 (2018).10.1109/CVPR.2018.00577 ↗
23. Zhu, M., Huang, J. & Guo, X. Image inpainting by end-to-end cascaded refinement with mask awareness. IEEE Trans. Image Process. 30, 4855–4866 (2021).10.1109/TIP.2021.3076310 ↗
24. Li, J., Wang, N., Zhang, L., Du, B. & Tao, D. Recurrent feature reasoning for image inpainting. In Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7757–7765 (2020).10.1109/CVPR42600.2020.00778 ↗
25. Yang, J., Qi, Z. & Shi, Y. Learning to incorporate structure knowledge for image inpainting. AAAI Conf. Artif. Intell. 34, 12605–12612 (2020).10.1609/aaai.v34i07.6951 ↗
26. Chen, Y., Xia, R., Yang, K. & Zou, K. MFMAM: image inpainting via multi-scale feature module with attention module. Comput. Vis. Image Underst. 238, 103883 (2024).10.1016/j.cviu.2023.103883 ↗
27. Zeng, Y., Fu, J., Chao, H. & Guo, B. Learning pyramid-context encoder network for high-quality image inpainting. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1486–1494 (2019).10.1109/CVPR.2019.00158 ↗
28. Chen, Y., Xia, R., Yang, K. & Zou, K. DNNAM: image inpainting algorithm via deep neural networks and attention mechanism. Appl. Soft Comput. 154, 111392 (2024).10.1016/j.asoc.2024.111392 ↗
29. Xie, C., Lu, J. & Lin, S. Image inpainting with learnable bidirectional attention maps. In Proc. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 8857–8866 (2019).10.1109/ICCV.2019.00895 ↗
30. Zheng, C., Cham, T.-J. & Cai, J. Pluralistic image completion. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1438–1447 (2019).10.1109/CVPR.2019.00153 ↗
31. Chen, Y., Xia, R., Yang, K. & Zou, K. MICU: image super-resolution via multi-level information compensation and U-net. Expert Syst. Appl. 245, 123111 (2024).10.1016/j.eswa.2023.123111 ↗
32. Zhang, W. et al. Fully context-aware image inpainting with a learned semantic pyramid. Pattern Recognit. 143, 109741 (2023).10.1016/j.patcog.2023.109741 ↗
33. Cao, J. et al. Ancient mural restoration based on a modified generative adversarial network. Herit. Sci. 8, 7 (2020).10.1186/s40494-020-0355-x ↗
34. Lv, C. et al. SeparaFill: two generators connected mural image restoration based on generative adversarial network with skip connect. Herit. Sci. 10, 135 (2022).10.1186/s40494-022-00771-w ↗
35. Merizzi, F. et al. Deep image prior inpainting of ancient frescoes in the Mediterranean Alpine arc. Herit. Sci. 12, 41 (2024).10.1186/s40494-023-01116-x ↗
36. Li, L. et al. Line drawing guided progressive inpainting of mural damages. Preprint at https://arxiv.org/abs/2211.06649 (2022).
37. Song, Z., Xuan, W., Liu, J., Li, Y. & Cao, L. Image restoration of Dun Huang murals based on auto-encoder generative adversarial neural network in Advanced Graphic Communication, Printing and Packaging Technology (eds Zhao, P. et al.) Lecture Notes in Electrical Engineering, Vol. 600 (Springer, 2020).10.1007/978-981-15-1864-5_26 ↗
38. Ma, L. et al. Toward fast, flexible, and robust low-light image enhancement. In Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5627–5636 (2022).10.1109/CVPR52688.2022.00555 ↗
39. Guo, C. et al. Zero-reference deep curve estimation for low-light image enhancement. In Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1777–1786 (2020).10.1109/CVPR42600.2020.00185 ↗
40. Fan, J., Yang, J., Wipf, D., Chen, B. & Tong, X. Image smoothing via unsupervised learning. ACM Trans. Graph. 37, 1–14 (2018).10.1145/3272127.3275081 ↗
41. Quan, W. et al. Image inpainting with local and global refinement. IEEE Trans. Image Process. 31, 2405–2420 (2022).10.1109/TIP.2022.3152624 ↗
42. Miyato, T., Kataoka, T., Koyama, M. & Yoshida, Y. Spectral normalization for generative adversarial networks. Preprint at https://arxiv.org/abs/1802.05957 (2018).
43. Liu, G. et al. Image inpainting for irregular holes using partial convolutions. In Proc. European Conference on Computer Vision, 85–100 (2018).10.1007/978-3-030-01252-6_6 ↗
44. Wang, N., Wang, W., Hu, W., Fenster, A. & Li, S. Thanka mural inpainting based on multi-scale adaptive partial convolution and stroke-like mask. IEEE Trans. Image Process. 30, 3720–3733 (2021).10.1109/TIP.2021.3064268 ↗
45. Johnson, J., Alahi, A. & Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Proc. European Conference on Computer Vision, 694–711 (2016).10.1007/978-3-319-46475-6_43 ↗
46. Gatys, L. A., Ecker, A. S. & Bethge, M. Image style transfer using convolutional neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2414–2423 (2016).10.1109/CVPR.2016.265 ↗
47. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at https://arxiv.org/abs/1409.1556 (2014).