A Review of Image Inpainting Methods Based on Deep Learning
Source abstract
Abstract
Image Inpainting is an age-old image processing problem, with people from different eras attempting to solve it using various methods. Traditional image inpainting algorithms have the ability to repair minor damage such as scratches and wear. However, with the rapid development of deep learning in the field of computer vision in recent years, coupled with abundant computing resources, methods based on deep learning have increasingly highlighted their advantages in semantic feature extraction, image transformation, and image generation. As such, image inpainting algorithms based on deep learning have become the mainstream in this domain.In this article, we first provide a comprehensive review of some classic deep-learning-based methods in the image inpainting field. Then, we categorize these methods based on component optimization, network structure design optimization, and training method optimization, discussing the advantages and disadvantages of each approach. A comparison is also made based on public datasets and evaluation metrics in image inpainting. Furthermore, the article delves into the applications of current image inpainting technologies, categorizing them into three major scenarios: object removal, general image repair, and facial inpainting. Finally, current challenges and prospective developments in the field of image inpainting are discussed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · AI in cancer detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221008, China
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
Funding
National Natural Science Foundation of China · Shanxi Provincial People’s Government · Fundamental Research Funds for the Central Universities · Ministry of Education
References (100+)
Show source-provided references
1. Miao, Y.R. (2020). Research on Spine Tumor CT Image Inpainting Method Based on Deep Learning. [Ph.D. Thesis, University of Chinese Academy of Sciences (Shenzhen Institutes of Advanced Technology, CAS)].
2. Zhao, M.Y. (2016). Research on Cloud Removal Methods for Remote Sensing Images. [Ph.D. Thesis, Tianjin University of Science & Technology].
3. Aerial Image Thick Cloud Inpainting Based on Improved Criminisi Algorithm
4. Dong, X.Y. (2021). Extraction of Architectural Objects and Recovery of Occlusion Information in Slant Remote Sensing Images. [Ph.D. Thesis, Harbin Engineering University].
5. Kriging Inpainting of Mountain Shadow Loss in Peak Cluster Depression Remote Sensing Image
6. Yang, Y. (2019). Lafin: Generative landmark guided face inpainting. arXiv.10.1007/978-3-030-60633-6_2 ↗
7. Shen, Z., Lai, W.S., Xu, T., Kautz, J., and Yang, M.H. (2018, January 18–23). Deep semantic face deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00862 ↗
8. DE-GAN: Domain Embedded GAN for High Quality Face Image Inpainting10.1016/j.patcog.2021.108415 ↗
9. Lahiri, A., Jain, A.K., and Agrawal, S. (2020, January 13–19). Prior guided gan based semantic inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01371 ↗
10. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. (2000, January 23–28). Image inpainting. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA.10.1145/344779.344972 ↗
11. Region Filling and Object Removal by Exemplar-Based Image Inpainting10.1109/TIP.2004.833105 ↗
12. Mathematical Models for Local Nontexture Inpaintings10.1137/S0036139900368844 ↗
13. Grossauer, H. (2004). European Conference on Computer Vision, Springer.
14. Scene completion using millions of photographs10.1145/1276377.1276382 ↗
15. Li, J., He, F., Zhang, L., Du, B., and Tao, D. (November, January 27). Progressive reconstruction of visual structure for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
16. Liao, L., Xiao, J., Wang, Z., Lin, C.-W., and Satoh, S. (2020). European Conference on Computer Vision, Springer.
17. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., and Ermon, S. (2021, January 4). Sdedit: Guided image synthesis and editing with stochastic differential equations. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
18. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., and Chen, M. (2021). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv.
19. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27–30). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.278 ↗
20. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., and Ebrahimi, M. (2019, January 27–28). Edgeconnect: Structure guided image inpainting using edge prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea.10.1109/ICCVW.2019.00408 ↗
21. Cai, J., Li, C., Tao, X., and Tai, Y.W. (2022, January 18–24). Image Multi-Inpainting via Progressive Generative Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00111 ↗
22. Zheng, C., Cham, T.-J., and Cai, J. (2019, January 15–20). Pluralistic image completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00153 ↗
23. Generative adversarial networks10.1145/3422622 ↗
24. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
25. Attention is all you need
26. Chu, T.H. (2021). Research on Image Inpainting and Recognition Methods of Contaminated License Plates Based on Machine Vision. [Ph.D. Thesis, North China University of Technology].
27. Scratch Detection and Removal Methods for Coal Rock Microscopic Images
28. Image Inpainting Method for Whole Well Electrical Imaging in Complex Stratum
29. Lv, C. (2017). Research on Removal and Inpainting Algorithm of Digital Image Mirror Reflection. [Ph.D. Thesis, Xiamen University].
30. Li, S. (2018). Research on Mobile Robot Semantic Map Building System. [Ph.D. Thesis, Beijing University of Technology].
31. Zheng, C. (2006). Research on Vision Road Detection and Tracking Algorithm for Micro-Robots. [Ph.D. Thesis, Nanjing University of Science & Technology].
32. Li, Y. (2021). Gait Recognition under Occlusion Based on Deep Learning. [Ph.D. Thesis, Harbin Engineering University].
33. Automatic Image Inpainting Algorithm for Apple Picking Robot Vision Based on LBM
34. Application Research of Image Inpainting Technology in Environmental Art Design
35. Liu, J. (2010). Research on Ancient Mural Image Protection and Intelligent Inpainting Technology. [Ph.D. Thesis, Zhejiang University].
36. A Review of Digital Inpainting Methods for Dunhuang Murals
37. Li, X. (2014). Research on Virtual Inpainting Technology for Ancient Murals. [Ph.D. Thesis, Xi’an University of Architecture and Technology].
38. Application of Digital Image Inpainting Technology in Cultural Relic Protection
39. Li, C. (2015). Automatic Marking and Virtual Inpainting of Mud Spots Diseases on Ancient Murals. [Ph.D. Thesis, Xi’an University of Architecture and Technology].
40. Duan, Y. (2021). Research on Irregular Interference Inpainting Algorithm for Ancient Stone Carved Documents. [Ph.D. Thesis, Kunming University of Science and Technology].
41. Exploration of Image Inpainting Algorithm Based on Maijishan Grottoes Relics
42. Research on Digital Image Inpainting Technology of Tibetan Murals Based on CDD Model
43. Places: A 10 million image database for scene recognition10.1109/TPAMI.2017.2723009 ↗
44. Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep learning face attributes in the wild. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.425 ↗
45. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
46. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
47. What makes paris look like paris?10.1145/2185520.2185597 ↗
48. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv.
49. Survey on deep learning image inpainting methods
50. Image Inpainting: A Review10.1007/s11063-019-10163-0 ↗
51. Image inpainting based on deep learning: A review10.1016/j.displa.2021.102028 ↗
52. A Survey on Image Inpainting Research Progress
53. Overview of Image Inpainting and Forensic Technology10.1155/2022/9403986 ↗
54. Ul Hassan, M. (2023, August 28). Alexnet Imagenet Classification with Deep Convolutional Neural Networks. Available online: https://neurohive.io/en/popular-networks/alexnet-imagenet-classification-with-deep-convolutional-neural-networks/.
55. Yu, F., and Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv.
56. Globally and locally consistent image completion10.1145/3072959.3073659 ↗
57. Zeng, Y., Lin, Z., Lu, H., and Patel, V.M. (2021, January 11–17). Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01390 ↗
58. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8–14). Image inpainting for irregular holes using partial convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01252-6_6 ↗
59. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (November, January 27). Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
60. Xiao, Q., Li, G., and Chen, Q. (2018). Deep inception generative network for cognitive image inpainting. arXiv.
61. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
62. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
63. Aggregated Contextual Transformations for High-Resolution Image Inpainting10.1109/TVCG.2022.3156949 ↗
64. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
65. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., and Li, H. (2017, January 21–26). High-resolution image inpainting using multi-scale neural patch synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.434 ↗
66. Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., and Kuo, C.C.J. (2018, January 8–14). Contextual-based image inpainting: Infer, match, and translate. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_1 ↗
67. Yan, Z., Li, X., Li, M., Zuo, W., and Shan, S. (2018, January 8–14). Shift-net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_1 ↗
68. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
69. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18–23). Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00577 ↗
70. Mohite, T.A., and Phadke, G.S. (2020, January 10–12). Image inpainting with contextual attention and partial convolution. Proceedings of the 2020 International Conference on Artificial Intelligence and Signal Processing (AISP), Amaravati, India.10.1109/AISP48273.2020.9073008 ↗
71. Xie, C., Liu, S., Li, C., Cheng, M.M., Zuo, W., Liu, X., and Ding, E. (November, January 27). Image inpainting with learnable bidirectional attention maps. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
72. Sagong, M.C., Shin, Y.G., Kim, S.W., Park, S., and Ko, S.J. (2019, January 15–20). Pepsi: Fast image inpainting with parallel decoding network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01162 ↗
73. Pepsi++: Fast and lightweight network for image inpainting10.1109/TNNLS.2020.2978501 ↗
74. Liu, H., Jiang, B., Xiao, Y., and Yang, C. (November, January 27). Coherent semantic attention for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
75. Image Inpainting Based on Inside–Outside Attention and Wavelet Decomposition10.1109/ACCESS.2020.2984587 ↗
76. Deep Image Inpainting with Enhanced Normalization and Contextual Attention10.1109/TCSVT.2022.3175171 ↗
77. Dual-Pyramidal Image Inpainting with Dynamic Normalization10.1109/TCSVT.2022.3165587 ↗
78. Wang, N., Li, J., Zhang, L., and Du, B. (2019, January 10–16). MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/520 ↗
79. Spatially adaptive multi-scale contextual attention for image inpainting10.1007/s11042-022-12489-9 ↗
80. Liu, H., Jiang, B., Song, Y., Huang, W., and Yang, C. (2020, January 23–28). Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58536-5_43 ↗
81. Zeng, Y., Fu, J., Chao, H., and Guo, B. (2019, January 15–20). Learning pyramid-context encoder network for high-quality image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00158 ↗
82. Zheng, C., Cham, T.J., Cai, J., and Phung, D. (2022, January 18–24). Bridging Global Context Interactions for High-Fidelity Image Completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01122 ↗
83. Zeng, Y., Lin, Z., Yang, J., Zhang, J., Shechtman, E., and Lu, H. (2020). European Conference on Computer Vision, Springer.
84. Yi, Z., Tang, Q., Azizi, S., Jang, D., and Xu, Z. (2020, January 13–19). Contextual residual aggregation for ultra high-resolution image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00753 ↗
85. Semantic-SCA: Semantic Structure Image Inpainting with the Spatial-Channel Attention10.1109/ACCESS.2021.3051982 ↗
86. Image inpainting with local and global refinement10.1109/TIP.2022.3152624 ↗
87. SIFNet: Free-form image inpainting using color split-inpaint-fuse approach10.1016/j.cviu.2022.103446 ↗
88. Deep image inpainting via contextual modelling in ADCT domain10.1049/ipr2.12590 ↗
89. Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., and Lempitsky, V. (2022, January 4–8). Resolution-robust large mask inpainting with fourier convolutions. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV51458.2022.00323 ↗
90. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., and Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
91. Dong, Q., Cao, C., and Fu, Y. (2022, January 18–24). Incremental transformer structure enhanced image inpainting with masking positional encoding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01107 ↗
92. Liu, Q., Tan, Z., Chen, D., Chu, Q., Dai, X., Chen, Y., and Yu, N. (2022, January 18–24). Reduce Information Loss in Transformers for Pluralistic Image Inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01106 ↗
93. Li, W., Lin, Z., Zhou, K., Qi, L., Wang, Y., and Jia, J. (2022, January 18–24). MAT: Mask-Aware Transformer for Large Hole Image Inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01049 ↗
94. Cao, C., Dong, Q., and Fu, Y. (2022). European Conference on Computer Vision, Springer.
95. Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., and Ebrahimi, M. (2019). Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv.
96. Ren, Y., Yu, X., Zhang, R., Li, T.H., Liu, S., and Li, G. (November, January 27). Structureflow: Image inpainting via structure-aware appearance flow. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
97. ECNFP: Edge-constrained network using a feature pyramid for image inpainting10.1016/j.eswa.2022.118070 ↗
98. Xiong, W., Yu, J., Lin, Z., Yang, J., Lu, X., Barnes, C., and Luo, J. (2019, January 15–20). Foreground-aware image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00599 ↗
99. Learning to Incorporate Structure Knowledge for Image Inpainting
100. Yamashita, Y., Shimosato, K., and Ukita, N. (2022, January 18–24). Boundary-Aware Image Inpainting with Multiple Auxiliary Cues. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00077 ↗