journal article·2024-03-12Open access

Weighted Similarity-Confidence Laplacian Synthesis for High-Resolution Art Painting Completion

Irawati Nurmala Sari, Weiwei Du

Applied Sciences· Volume 14· Issue 6

Abstract

Artistic image completion assumes a significant role in the preservation and restoration of invaluable art paintings, marking notable advancements through the adoption of deep learning methodologies. Despite progress, challenges persist, particularly in achieving optimal results for high-resolution paintings. The intricacies of complex structures and textures in art paintings pose difficulties for sophisticated approaches like Generative Adversarial Networks (GANs), leading to issues such as small-scale texture synthesis and the inference of missing information, resulting in distortions in lines and unnatural colors. Simultaneously, patch-based image synthesis, augmented with global optimization on the image pyramid, has evolved to enhance structural coherence and details. However, gradient-based synthesis methods face obstacles related to directionality, inconsistency, and the computational burdens associated with solving the Poisson equation in non-integrable gradient fields. This paper introduces a pioneering approach, integrating Weighted Similarity-Confidence Laplacian Synthesis to comprehensively address these challenges and advance the field of artistic image completion. Experimental results affirm the effectiveness of our approach, offering promising outcomes for the preservation and restoration of art paintings with intricate details and irregular missing regions. The integration of weighted Laplacian synthesis and patch-based completion across multi-regions ensures precise and targeted completion, outperforming existing methods. A comparative analysis underscores our method’s superiority in artifact reduction and minimizing blurriness, particularly addressing challenges related to color discrepancies in texture areas. Additionally, the incorporation of pyramid blending proves advantageous, ensuring smoother transitions and preventing noticeable seams or artifacts in blended results. Based on empirical results, our method consistently outperforms previous methods across both high and low resolutions. Responding to these insights, our approach emerges as an invaluable guide for both curators and artists. The algorithm’s performance yields insights that underscore the central role of thoughtful decision making in the creation of art paintings. This guidance extends to informing choices related to color selection, brushstrokes, and various other elements integral to the artistic process. During the creation phase, employing these insights enables artists and curators to optimize not only the digitization but also the subsequent restoration process. This proves especially vital when dealing with the intricacies involved in physically restoring damaged original art paintings. Importantly, our approach not only streamlines the restoration process but also contributes significantly to the preservation and enhancement of the digital representations of these distinctive and often intricate works of art.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Aesthetic Perception and Analysis · Image and Signal Denoising Methods

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Informatics Engineering, Faculty of Computer Science, Brawijaya University, Malang 65145, Indonesia

Information and Human Science, Kyoto Institute of Technology, Kyoto 606-8585, Japan

References (37)

Show source-provided references
1. Criminisi, A., Pérez, P., and Toyama, K. (July, January 27). Object removal by exemplar-based inpainting. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.
2. Efros, A.A., and Leung, T.K. (1999, January 20–25). Texture synthesis by non-parametric sampling. Proceedings of the 7th IEEE International Conference on Computer Vision, Corfu, Greece.10.1109/ICCV.1999.790383 ↗
3. Image melding: Combining inconsistent images using patch-based synthesis10.1145/2185520.2185578 ↗
4. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18–23). Generative Image Inpainting with Contextual Attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00577 ↗
5. Artistic Style Transfer for Image Inpainting
6. Zeng, Y., Lin, Z., Yang, J., Zhang, J., Shechtman, E., and Lu, H. (2020, January 23–28). High-resolution image inpainting with iterative confidence feedback and guided upsampling. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58529-7_1 ↗
7. Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., and Kuo, C.C.J. (2018). Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv.
8. Yi, Z., Tang, Q., Azizi, S., Jang, D., and Xu, Z. (2020, January 14–19). Contextual residual aggregation for ultra high-resolution image inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00753 ↗
9. Globally and locally consistent image completion10.1145/3072959.3073659 ↗
10. Ma, Y., Liu, X., Bai, S., Wang, L., Liu, A., Tao, D., and Hancock, E. (2019). Region-wise generative adversarial image inpainting for large missing areas. arXiv.
11. Li, J., Wang, N., Zhang, L., Du, B., and Tao, D. (2020, January 13–19). Recurrent feature reasoning for image inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00778 ↗
12. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8–14). Image inpainting for irregular holes using partial convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01252-6_6 ↗
13. An image inpainting technique based on the fast marching method10.1080/10867651.2004.10487596 ↗
14. A generalized accelerated proximal gradient approach for total-variation-based image restoration10.1109/TIP.2011.2131665 ↗
15. Algorithms and software for total variation image reconstruction via first-order methods10.1007/s11075-009-9310-3 ↗
16. Nontexture inpainting by curvature-driven diffusions10.1006/jvci.2001.0487 ↗
17. Total variation denoising and enhancement of color images based on the CB and HSV color models10.1006/jvci.2001.0491 ↗
18. Irawati, N.S., Urano, Y., and Du, W. (2021, January 20–22). Image inpainting using orthogonal viewpoints and structure consistency in Manhattan World. Proceedings of the 8th International Virtual Conference on Applied Computing and Information Technology (ACIT), Kanazawa, Japan.
19. Horikawa, E., Irawati, N.S., and Du, W. (2021, January 20–22). Image inpainting using clustered planar structure guidance. Proceedings of the 8th International Virtual Conference on Applied Computing and Information Technology (ACIT), Kanazawa, Japan.10.1145/3468081.3471133 ↗
20. Interactive Image Inpainting of Large-Scale Missing Region10.1109/ACCESS.2021.3072366 ↗
21. Structure-Texture Consistent Painting Completion for Artworks10.1109/ACCESS.2023.3252892 ↗
22. Urano, Y., Sari, I.N., and Du, W. (2022, January 4–7). Image inpainting using automatic structure propagation with auxiliary line construction. Proceedings of the 23rd ACIS International Summer Virtual Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD Summer), Kyoto, Japan.10.1109/SNPD-Summer57817.2022.00026 ↗
23. Takarabe, J., Sari, I.N., and Du, W. (July, January 30). Depth map estimation of single-view image using smartphone camera for a 3 dimension image generation in augmented reality. Proceedings of the 6th International Symposium on Computer, Consumer and Control (IS3C), Taichung, Taiwan.
24. Masaoka, K., Sari, I.N., and Du, W. (July, January 30). Vanishing points detection with line segments of gaussian sphere. Proceedings of the 6th International Symposium on Computer, Consumer and Control (IS3C), Taichung, Taiwan.
25. Sari, I.N., Masaoka, K., Takarabe, J., and Du, W. (July, January 30). High-resolution art painting completion using multi-region laplacian fusion. Proceedings of the 6th International Symposium on Computer, Consumer and Control (IS3C), Taichung, Taiwan.
26. Masaoka, K., Sari, I.N., and Du, W. (2022, January 4–7). Edge-enhanced GAN with vanishing points for image inpainting. Proceedings of the IEEE/ACIS International Summer Virtual Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kyoto City, Japan.10.1109/SNPD-Summer57817.2022.00027 ↗
27. Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., and Ebrahimi, M. (2019). Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv.
28. Region filling and object removal by exemplar-based image inpainting10.1109/TIP.2004.833105 ↗
29. Image completion with structure propagation10.1145/1073204.1073274 ↗
30. Image inpainting for digital dunhuang murals using partial convolutions and sliding window method10.1088/1742-6596/1302/3/032040 ↗
31. Wang, H., Li, Q., and Zou, Q. (2018). Inpainting of dunhuang durals by sparsely modeling the texture similarity and structure continuity. J. Comput. Cult. Herit.10.1145/3280790 ↗
32. Lee, J.H., Choi, I., and Kim, M.H. (July, January 26). Laplacian patch-based image synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
33. Anmol, B., Green, R.K.S., and Sachin, D.L. (2020). Spatially variant laplacian pyramids for multi-frame exposure fusion. Comput. Vis. Image Process., 73–81.10.1007/978-981-15-4015-8_7 ↗
34. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
35. Ding, K., Ma, K., Wang, S., and Simoncell, E.E. (2020, January 13–19). Image quality assessment: Unifying structure and texture similarity. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/TPAMI.2020.3045810 ↗
36. Wang, Z., Bovik, A.C., and Lu, L. (2002, January 13–17). Why is image quality assessment so difficult. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Orlando, FL, USA.10.1109/ICASSP.2002.5745362 ↗
37. A critical survey of state-of-the-art image inpainting quality assessment metric10.1016/j.jvcir.2017.09.006 ↗