Multi-level feature fusion and high-fidelity information transfer algorithm for mural inpainting
Source abstract
Abstract
This paper proposes a multi-level feature fusion and high-fidelity information transfer algorithm (MFIT) to enhance the quality of mural inpainting. This method incorporates a Feature Transfer Module (FTM) designed to optimize the information flow, thereby facilitating the seamless integration of high-level semantic features with the complex low-level structural details. Moreover, we introduce gated convolutions in conjunction with recursive mechanisms in a parallel architecture to form a Recursive Gated Network (RGNet), which enables arbitrary-order spatial information exchange across multi-scale features. Finally, we integrate a Mask-Aware Pixel-Shuffle Down-Sampling Module (MPD) into an unquantized Transformer network, which effectively preserves critical information in occluded regions by exploiting spatial context within masked areas. Experimental results reveal that, compared to the baseline model, the proposed approach on the Dunhuang mural dataset achieves improvements in PSNR by 1.40% ~ 5.49%, SSIM by 0.43% ~ 1.65%, a reduction in L1 by 6.25% ~ 15.63%, and a decrease in LPIPS by 2.47% ~ 17.46%.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Computer Graphics and Visualization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (31)
Show source-provided references
1. Chen, Y., Tao, M. & Zhao, M. Generative adversarial mural restoration based on structural gated and texture joint guidance. J. Hunan Univ. Nat. Sci. 50, 1–11 (2022).
2. Wang, H., Li, Q. & Jia, S. A global and local feature weighted method for ancient murals inpainting. Int. J. Mach. Learn. Cybern. 11, 1197–1216 (2020).10.1007/s13042-019-01032-2 ↗
3. Cao, R. C. et al. Generative image inpainting with attention propagation. Acta Autom. Sin. 48, 1343–1352 (2022).
4. Sridevi, G. & Srinivas Kumar, S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circ. Syst. Signal Process. 38, 3802–3817 (2019).10.1007/s00034-019-01029-w ↗
5. Qiang, Z., He, L. & Xu, D. Exemplar-based pixel by pixel inpainting based on patch shift. In Computer Vision: Second CCF Chinese Conference, CCCV 2017, 370–382 (Springer Singapore, 2017).10.1007/978-981-10-7302-1_31 ↗
6. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012).
7. Cortes, C. et al. Advances in neural information processing systems 28. Proc. 29th Annu. Conf. Neural Inf. Process. Syst. (2015).
8. Dosovitskiy, A. et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In Proceedings of the 9th International Conference on Learning Representations (2021).
9. Zhou, Y. et al. TransFill: reference-guided image inpainting by merging multiple color and spatial transformations. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 2266-2276 (2021).10.1109/CVPR46437.2021.00230 ↗
10. Li, W. et al. MAT: mask-aware transformer for large hole image inpainting. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 10758–10768 (2022).10.1109/CVPR52688.2022.01049 ↗
11. Dong, Q., Cao, C. & Fu, Y. Incremental transformer structure enhanced image inpainting with masking positional encoding. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 11358–11368 (IEEE, 2022).10.1109/CVPR52688.2022.01107 ↗
12. Deng, Y. et al. T-former: an efficient transformer for image inpainting. Proc. 30th ACM Int. Conf. Multimedia. 6559–6568 (ACM, 2022).10.1145/3503161.3548446 ↗
13. Chen, S., Atapour-Abarghouei, A. & Shum, H. P. H. HINT: high-quality inpainting transformer with mask-aware encoding and enhanced attention. IEEE Trans. Multimedia (2024).10.1109/TMM.2024.3369897 ↗
14. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (2014).
15. Walker, J. et al. An uncertain future: forecasting from static images using variational autoencoders. In Computer Vision – ECCV 2016 835–851 (Springer, Cham, 2016).10.1007/978-3-319-46478-7_51 ↗
16. Burda, Y., Grosse, R. & Salakhutdinov, R. Importance weighted autoencoders. In Proceedings of the 4th International Conference on Learning Representations (2016).
17. Van Den Oord, A. & Vinyals, O. Neural discrete representation learning. Adv. Neural Inf. Process. Syst. 30, 6306–6315 (2017).
18. Esser, P., Rombach, R. & Ommer, B. Taming transformers for high-resolution image synthesis. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 12873–12883 (2021).10.1109/CVPR46437.2021.01268 ↗
19. Ramesh, A. et al. Zero-shot text-to-image generation. In Proc. 38th Int. Conf. Mach. Learn. 8821-8831 (PMLR, 2021).
20. Tian, H. et al. Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting. Comput. Vis. Media. 9, 827–841 (2023).10.1007/s41095-022-0331-3 ↗
21. Liu, Q. et al. Transformer based pluralistic image completion with reduced information loss. IEEE Trans. Pattern Anal. Mach. Intell. 46, 123–135 (2024).10.1109/TPAMI.2024.3384406 ↗
22. Yu, J. et al. Generative image inpainting with contextual attention. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 5505-5514 (IEEE, 2018).10.1109/CVPR.2018.00577 ↗
23. Yu, J. et al. Free-form image inpainting with gated convolution. Proc. IEEE/CVF Int. Conf. Comput. Vis. 4471––4480 (IEEE, 2019).10.1109/ICCV.2019.00457 ↗
24. Shi, W. et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 1874-1883 (IEEE, 2016).10.1109/CVPR.2016.207 ↗
25. Chollet, F. Xception: Deep learning with depthwise separable convolutions. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 1251–1258 (IEEE, 2017).10.1109/CVPR.2017.195 ↗
26. Liu, G. et al. Image inpainting for irregular holes using partial convolutions. Proc. Eur. Conf. Comput. Vis. 85–100 (Springer, 2018).10.1007/978-3-030-01252-6_6 ↗
27. Yu, T. et al. Dunhuang Grottoes painting dataset and benchmark. Herit. Sci. 8, 15 (2020).10.1186/s40494-020-0353-z ↗
28. Quan, W. et al. Image inpainting with local and global refinement. IEEE Trans. Image Process. 31, 2405–2420 (2022).10.1109/TIP.2022.3152624 ↗
29. Li, X. et al. MISF: Multi-level interactive siamese filtering for high-fidelity image inpainting. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 1234–1243 (IEEE, 2022).10.1109/CVPR52688.2022.00191 ↗
30. Lin, J. et al. Multi-feature co-learning for image inpainting. Proc. 26th Int. Conf. Pattern Recognit. 296–302 (IEEE, 2022).10.1109/ICPR56361.2022.9956475 ↗
31. Karras, T., Laine, S. & Aila, T. A style-based generator architecture for generative adversarial networks. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 4401––4410 (IEEE, 2019).10.1109/CVPR.2019.00453 ↗