A Context-Based Multi-Scale Discriminant Model for Natural Image Inpainting
Source abstract
Abstract
Image inpainting is a classical yet challenging problem in computer vison with many real-world applications. In this paper, we propose a novel method for inpainting based on Convolutional Neural Network (CNN) and Generative Adversarial Networks (GAN). By analogy with the autoencoder, a new context encoder is proposed to generate the contents of a missing image region conditioned on its surroundings. To improve the stability and efficiency, we also modify the architecture of the context encoder by introducing 1*1 convolution, as well as other improvements. In addition, a multi-scale discriminator combining with GAN is presented. During training, the reconstruction and adversarial losses are used for the global image. To generate more details, we further add a local texture loss for the missing part. Qualitative experiments show that our model performs favorably against classical methods in generating visually plausible inpainting results. Moreover, quantitative experiments substantiate the effectiveness of the proposed method in natural image inpainting.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
China University of Petroleum
References (45)
Show source-provided references
1. Maas A. L. · International Conference on Machine Learning (ICML) · 2013 · 30 · 3
2. Bertalmio M., Bertozzi A L, Sapiro G. Navier-Stokes, fluid dynamics, and image and video inpainting. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2001, I-I.
3. Barnes C · ACM Transactions on Graphics · 2009 · 2810.1145/1531326.1531330 ↗
4. 10.1109/ICCV.2011.612627810.1109/ICCV.2011.6126278 ↗
5. Rumelhart D.E. · Cognitive Modeling · 1988 · 5 · 1
6. 10.1162/neco.1989.1.4.54110.1162/neco.1989.1.4.541 ↗
7. LeCun Y. · Proceedings of the IEEE · 1986 · 1986 · 227810.1109/5.726791 ↗
8. 10.1109/CVPR.2014.8110.1109/CVPR.2014.81 ↗
9. Krizhevsky A., Sutskever I., Hinton G. E. ImageNet classification with deep convolutional neural networks. Proceedings of the International Conference on Neural Information Processing Systems (NIPS) 2012, 1097–1105.
10. 10.1109/TPAMI.2016.257268310.1109/TPAMI.2016.2572683 ↗
11. 10.1109/CVPR.2015.729893510.1109/CVPR.2015.7298935 ↗
12. 10.1109/CVPR.2016.26510.1109/CVPR.2016.265 ↗
13. 10.1109/CVPR.2016.26510.1109/CVPR.2016.265 ↗
14. Ren J.S., Xu L., Yan Q. Shepard convolutional neural networks. Proceedings of the International Conference on Neural Information Processing Systems (NIPS) 2015, 901–909.
15. 10.1145/1390156.139029410.1145/1390156.1390294 ↗
16. Kingma D.P. · arXiv · 2013 · 1312 · 6114
17. Larsen A., Sønderby S., Winther O. Autoencoding beyond pixels using a learned similarity metric. International Conference on Machine Learning (ICML) 2016, 1558–1566.
18. Goodfellow I. J, Pouget-Abadie J., Mirza M. Generative adversarial Networks. Proceedings of the International Conference on Neural Information Processing Systems (NIPS) 2014, 2672–2680.
19. Mirza M., Osindero S. Conditional generative adversarial net. arXiv,1411.1784, 2014.
20. Arjovsky M., Bottou L. Towards principled methods for training generative adversarial networks. arXiv,1701.04862, 2017.
21. Radford A., Metz L., Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv, 1511.06434, 2015.
22. Berthelot D., Schumm T., Began Metz L.: Boundary equilibrium generative adversarial networks. arXiv, 1703.10717, 2017.
23. Arjovsky M., Chintala S., gan Bottou L. Wasserstein.arXiv, 1701.07875, 2017.
24. 10.1109/CVPR.2016.27810.1109/CVPR.2016.278 ↗
25. 10.1145/3072959.307365910.1145/3072959.3073659 ↗
26. 10.1007/978-3-030-01252-6_610.1007/978-3-030-01252-6_6 ↗
27. 10.1109/CVPR.2018.0057710.1109/CVPR.2018.00577 ↗
28. Yu J., Lin Z., Yang J. Free-form image inpainting with gated convolution. arXiv,1806.03589, 2018.10.1109/ICCV.2019.00457 ↗
29. 10.24963/ijcai.2019/52010.24963/ijcai.2019/520 ↗
30. Liu H., Jiang B., Xiao Y. Coherent Semantic Attention for Image Inpainting. arXiv, 1905.12384, 2019.10.1109/ICCV.2019.00427 ↗
31. Nazeri K., Ng E., Joseph T. EdgeConnect: Generative image inpainting with adversarial edge learning. arXiv, 1901.00212, 2019.
32. 10.1109/CVPR.2019.0059910.1109/CVPR.2019.00599 ↗
33. 10.1109/CVPR.2019.0015310.1109/CVPR.2019.00153 ↗
34. 10.1109/CVPR.2017.62410.1109/CVPR.2017.624 ↗
35. 10.1093/nq/s4-IX.211.39-a10.1093/nq/s4-IX.211.39-a ↗
36. 10.1007/978-3-030-01249-6_910.1007/978-3-030-01249-6_9 ↗
37. Ioffe S., Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv, 1502.03167, 2015.
38. Lin M. · arXiv · 2013 · 1312 · 4400
39. Kingma D.P., Ba J. Adam: A Method for Stochastic Optimization 2014, 1–15. http://arxiv.org/abs/1412.6980.
40. Deng J., Berg A., Satheesh S.ILSVRC-2012, 2012. http://www.image-net.org/challenges/LSVRC.
41. 10.1109/CVPR.2016.12410.1109/CVPR.2016.124 ↗
42. 10.1109/CVPR.2014.46110.1109/CVPR.2014.461 ↗
43. Yu F., Zhang Y., Song S. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. Computer Science, 2015.
44. Krizhevsky A., Nair V., Hinton G.E. The CIFAR-10 dataset. http://www.cs.toronto.edu.
45. Shao M., Wang C., Wu T. Context-Based Multiscale Unified Network for Missing Data Reconstruction in Remote Sensing Images. IEEE Geoscience and Remote Sensing Letters, 2020.