Highly accurate mural sketch generation for Dunhuang murals with enhanced patch-GAN
Source abstract
Abstract
Dunhuang murals are historical treasures, which have been eroded by various deteriorations during thousands of years. As a form of conservation, mural sketch drawing is widely used. However, the drawing is done manually by highly skilled experts, requiring high level of artistic appreciation and is time-consuming. A variety of algorithms for automatic mural sketch extraction were proposed. These algorithms either fail to filter the mural deteriorations effectively or barely obtain bold mural lines which are not enough for practical use. Motivated by these observations, we propose a novel deep generative model with edge guidance, stronger backbone, scaled cross-entropy and stronger supervision from multi-scale PatchGAN. With these simple yet effective enhancements, our proposed model can not only filter mural deterioration, but also generate thin lines, making this model a practical high-accuracy one. Experiments on the Dunhuang mural dataset and generalization study on non-Dunhuang murals demonstrate the effectiveness of our method.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (42)
Show source-provided references
1. Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986).10.1109/TPAMI.1986.4767851 ↗
2. He, J., Wang, S., Zhang, Y. & Zhang, J. A computational fresco sketch generation framework. In Proceeding IEEE International Conference on Multimedia and Expo (ICME) (IEEE, 2013).
3. Sun, D., Zhang, J., Pan, G. & Zhan, R. Mural2Sketch: a combined line drawing generation method for ancient mural painting. In Proceeding IEEE International Conference on Multimedia and Expo (ICME) (IEEE, 2018).10.1109/ICME.2018.8486504 ↗
4. Kittler, J. On the accuracy of the Sobel edge detector. Image Vis. Comput. 1, 37–42 (1983).10.1016/0262-8856(83)90006-9 ↗
5. Xu, L., Lu, C., Xu, Y. & Jia, J. Image smoothing via L0 gradient minimization. ACM Trans. Graph. 30, 1–12 (2011).
6. Li, L., Guo, X., Feng, W. & Zhang, J. Soft clustering guided image smoothing. In Proceeding IEEE International Conference on Multimedia and Expo (ICME) (IEEE, 2018).10.1109/ICME.2018.8486448 ↗
7. Soria, X., Riba, E. & Sappa, A. Dense extreme inception network: towards a robust CNN model for edge detection. In Proceeding IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2020).10.1109/WACV45572.2020.9093290 ↗
8. Nazeri, K., Ng, E., Joseph, T., Qureshi, F. & Ebrahimi, M. Edgeconnect: structure guided image inpainting using edge prediction. In Proceeding. IEEE International Conference on Computer Vision Workshops (ICCVW) 1912–1921 (IEEE, 2019).10.1109/ICCVW.2019.00408 ↗
9. Xie, S. & Tu, Z. Holistically-nested edge detection. In Proceeding. IEEE International Conference on Computer Vision (ICCV) 1395–1403 (IEEE, 2015).10.1109/ICCV.2015.164 ↗
10. Liu, Y. et al. Prior semantic-embedding representation learning for on-the-fly FG-SBIR. Expert Syst. Appl. 255, 124532 (2024).10.1016/j.eswa.2024.124532 ↗
11. Liu, Y., Dai, D., Wang, G. & Xia, S. Multivariate feedback-based image-text joint learning for sketch-less facial image retrieval. IEEE Trans. Circuits Syst. Video Technol. https://ieeexplore.ieee.org/document/11021462 (2025).10.1109/TCSVT.2025.3575732 ↗
12. Liu, B., Du, S., Li, J., Wang, J. & Liu, W. Dunhuang mural line drawing based on bi-dexined network and adaptive weight learning. In Proceeding Pattern Recognition and Computer Vision, PRCV 279–292 (Springer, 2022).10.1007/978-3-031-18907-4_22 ↗
13. Liu, B., He, F., Du, S., Zhang, K. & Wang, J. Dunhuang murals contour generation network based on convolution and self-attention fusion. Appl. Intell. 53, 22073–22085 (2023).10.1007/s10489-023-04614-4 ↗
14. Pan, G., Sun, D., Zhan, R. & Zhang, J. Mural sketch generation via style-aware convolutional neural network. In Proceeding Computer Graphics International 239–245 (ACM, 2018).10.1145/3208159.3208160 ↗
15. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceeding International Conference on Learning Representations (ICLR) (2015).
16. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2818–2826 (IEEE, 2016).10.1109/CVPR.2016.308 ↗
17. Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L. Semantic image segmentation with deep convolutional nets and fully connected CRFs. Computer Science, 357–361 (2014).
18. Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2017).10.1109/TPAMI.2017.2699184 ↗
19. Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceeding European Conference on Computer Vision (ECCV) 833–851 (Springer, 2018).10.1007/978-3-030-01234-2_49 ↗
20. Wang, J. et al. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3349–3364 (2020).10.1109/TPAMI.2020.2983686 ↗
21. Kirillov, A., Girshick, R., He, K. & Dollár, P. Panoptic feature pyramid networks. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 6392–6401 (IEEE, 2019).10.1109/CVPR.2019.00656 ↗
22. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In Proceeding Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Proceedings, Vol. 9351 (Springer, 2015).10.1007/978-3-319-24574-4_28 ↗
23. Qin, X. et al. U2-Net: going deeper with nested U-structure for salient object detection. In Proceeding Conference on Computer Vision and Pattern Recognition Vol. 106, 107404 (2020).10.1016/j.patcog.2020.107404 ↗
24. Goodfellow, I. et al. Generative adversarial networks. Commun. ACM 63, 139–144 (2020).10.1145/3422622 ↗
25. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V. & Courville, A. Improved training of Wasserstein GANs. In Proceeding Conference on Neural Information Processing Systems (NIPS) (ACM, 2017).
26. Karras, T., Aila, T., Laine, S. & Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation. In Proceeding International Conference on Learning Representations (ICLR) (2018).
27. Xu, T. et al. AttnGAN: fine-grained text to image generation with attentional generative adversarial networks. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1316–1324 (IEEE, 2018).10.1109/CVPR.2018.00143 ↗
28. Zhao, J., Mathieu, M. & LeCun, Y. Energy-based generative adversarial network. In Proceeding International Conference on Learning Representations (ICLR) (2017).
29. Vondrick, C., Pirsiavash, H. & Torralba, A. Generating videos with scene dynamics. In Proceeding Conference on Neural Information Processing Systems (NIPS) (ACM, 2016).
30. Dai, B., Fidler, S., Urtasun, R. & Lin, D. Towards diverse and natural image descriptions via a conditional GAN. In Proceeding IEEE International Conference on Computer Vision (ICCV) 2989–2998 (IEEE, 2017).10.1109/ICCV.2017.323 ↗
31. Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C. & Yang, Y. HoloGAN: unsupervised learning of 3D representations from natural images. In Proceeding IEEE International Conference on Computer Vision (ICCV) 7587–7596 (IEEE, 2019).10.1109/ICCV.2019.00768 ↗
32. Mathieu, M., Couprie, C. & LeCun, Y. Deep multi-scale video prediction beyond mean square error. In Proceeding International Conference on Learning Representations (ICLR) (2016).
33. Tulyakov, S., Liu, M. Y., Yang, X. & Kautz, J. MoCoGAN: decomposing motion and content for video generation. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1526–1535 (IEEE, 2018).10.1109/CVPR.2018.00165 ↗
34. Yu, J. et al. Generative image inpainting with contextual attention. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5505–5514 (IEEE, 2018).10.1109/CVPR.2018.00577 ↗
35. Yu, J. et al. Free-form image inpainting with gated convolution. In Proceeding IEEE International Conference on Computer Vision (ICCV) 4470–4479 (IEEE, 2019).10.1109/ICCV.2019.00457 ↗
36. Liu, W. et al. Fine-grained image inpainting with scale-enhanced generative adversarial network. Pattern Recognit. Lett. 143, 81–87 (2021).10.1016/j.patrec.2020.12.008 ↗
37. Mirza, M. & Osindero, S. Conditional generative adversarial nets. Computer Science, 2672–2680 (2014).
38. Isola, P., Zhu, J. Y., Zhou, T. & Efros A. Image-to-image translation with conditional adversarial networks. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5967–5976 (IEEE, 2017).10.1109/CVPR.2017.632 ↗
39. Wang, T. C. et al. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 8798-8807 (IEEE, 2018).10.1109/CVPR.2018.00917 ↗
40. Park, T., Liu, M. Y., Wang, T. C. & Zhu, J. Y. Semantic image synthesis with spatially-adaptive normalization. In Proceeding IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2332–2341 (IEEE, 2019).10.1109/CVPR.2019.00244 ↗
41. Yu, T. et al. Artificial intelligence for Dunhuang cultural heritage protection: the project and the dataset. Int. J. Comput. Vis. 130, 2646–2673 (2022).10.1007/s11263-022-01665-x ↗
42. Qin, X. et al. Highly accurate dichotomous image segmentation. In Proceeding European Conference on Computer Vision (ECCV) 38–56 (Springer, 2022).10.1007/978-3-031-19797-0_3 ↗