Video-Restoration-Net: Deep Generative Model with Non-Local Network for Inpainting and Super-Resolution Tasks
Source abstract
Abstract
Although deep learning-based approaches for video processing have been extensively investigated, the lack of generality in network construction makes it challenging for practical applications, particularly in video restoration. As a result, this paper presents a universal video restoration model that can simultaneously tackle video inpainting and super-resolution tasks. The network, called Video-Restoration-Net (VRN), consists of four components: (1) an encoder to extract features from each frame, (2) a non-local network that recombines features from adjacent frames or different locations of a given frame, (3) a decoder to restore the coarse video from the output of a non-local block, and (4) a refinement network to refine the coarse video on the frame level. The framework is trained in a three-step pipeline to improve training stability for both tasks. Specifically, we first suggest an automated technique to generate full video datasets for super-resolution reconstruction and another complete-incomplete video dataset for inpainting, respectively. A VRN is then trained to inpaint the incomplete videos. Meanwhile, the full video datasets are adopted to train another VRN frame-wisely and validate it against authoritative datasets. We show quantitative comparisons with several baseline models, achieving 40.5042 dB/0.99473 on PSNR/SSIM in the inpainting task, while during the SR task we obtained 28.41 dB/0.7953 and 27.25/0.8152 on BSD100 and Urban100, respectively. The qualitative comparisons demonstrate that our proposed model is able to complete masked regions and implement super-resolution reconstruction in videos of high quality. Furthermore, the above results show that our method has greater versatility both in video inpainting and super-resolution tasks compared to recent models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic Information, Wuhan University, Wuhan 430072, China
Funding
National Natural Science Foundation of China Enterprise Innovation and Development Joint Fund Key Project
References (52)
Show source-provided references
1. PatchMatch: A randomized correspondence algorithm for structural image editing10.1145/1531326.1531330 ↗
2. Scene completion using millions of photographs10.1145/1276377.1276382 ↗
3. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27–30). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.278 ↗
4. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., and Li, H. (2017, January 21–26). High-resolution image inpainting using multi-scale neural patch synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.434 ↗
5. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (November, January 27). Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
6. Liu, M., Chen, S., Liu, J., and Tang, X. (2009, January 19–24). Video completion via motion guided spatial-temporal global optimization. Proceedings of the 17th ACM International Conference on Multimedia, Beijing, China.10.1145/1631272.1631350 ↗
7. Virtual contour guided video object inpainting using posture mapping and retrieval10.1109/TMM.2010.2095000 ↗
8. Granados, M., Kim, K.I., Tompkin, J., Kautz, J., and Theobalt, C. (2012, January 7–13). Background inpainting for videos with dynamic objects and a free-moving camera. Proceedings of the Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy. Proceedings, Part I 12.10.1007/978-3-642-33718-5_49 ↗
9. Barnes, C., Shechtman, E., Goldman, D.B., and Finkelstein, A. (2010, January 5–11). The generalized patchmatch correspondence algorithm. Proceedings of the Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece. Proceedings, Part III 11.10.1007/978-3-642-15558-1_3 ↗
10. Temporally coherent completion of dynamic video
11. Video inpainting of complex scenes10.1137/140954933 ↗
12. Le, T.T., Almansa, A., Gousseau, Y., and Masnou, S. (2017, January 17–20). Motion-consistent video inpainting. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296651 ↗
13. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
14. Zhang, Z., Wang, Z., Lin, Z., and Qi, H. (2019, January 15–20). Image super-resolution by neural texture transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00817 ↗
15. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. (2018, January 18–23). Residual dense network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00262 ↗
16. Super-resolution without explicit subpixel motion estimation10.1109/TIP.2009.2023703 ↗
17. Dai, Q., Yoo, S., Kappeler, A., and Katsaggelos, A.K. (2015, January 27–30). Dictionary-based multiple frame video super-resolution. Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada.10.1109/ICIP.2015.7350764 ↗
18. Liao, R., Tao, X., Li, R., Ma, Z., and Jia, J. (2015, January 7–12). Video super-resolution via deep draft-ensemble learning. Proceedings of the IEEE International Conference on Computer Vision, Boston, MA, USA.10.1109/ICCV.2015.68 ↗
19. Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., and Shi, W. (2017, January 21–26). Real-time video super-resolution with spatio-temporal networks and motion compensation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.304 ↗
20. Video enhancement with task-oriented flow10.1007/s11263-018-01144-2 ↗
21. Tao, X., Gao, H., Liao, R., Wang, J., and Jia, J. (2017, January 21–26). Detail-revealing deep video super-resolution. Proceedings of the IEEE International Conference on Computer Vision, Honolulu, HI, USA.10.1109/ICCV.2017.479 ↗
22. Sajjadi, M.S., Vemulapalli, R., and Brown, M. (2018, January 18–23). Frame-recurrent video super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00693 ↗
23. Generative adversarial networks10.1145/3422622 ↗
24. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–23). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
25. Globally and locally consistent image completion10.1145/3072959.3073659 ↗
26. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18–23). Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00577 ↗
27. Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., and Kuo, C.C.J. (2018, January 8–14). Contextual-based image inpainting: Infer, match, and translate. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_1 ↗
28. Yan, Z., Li, X., Li, M., Zuo, W., and Shan, S. (2018, January 8–14). Shift-net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_1 ↗
29. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18.
30. Yang, J., Qi, Z., and Shi, Y. (2020, January 7–12). Learning to incorporate structure knowledge for image inpainting. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. No. 07.10.1609/aaai.v34i07.6951 ↗
31. Liu, H., Jiang, B., Song, Y., Huang, W., and Yang, C. (2020, January 23–28). Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part II 16.10.1007/978-3-030-58536-5_43 ↗
32. Wan, Z., Zhang, J., Chen, D., and Liao, J. (2021, January 11–17). High-fidelity pluralistic image completion with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Online Meeting.10.1109/ICCV48922.2021.00465 ↗
33. Masnou, S., and Morel, J.M. (1998, January 4–7). Level lines based disocclusion. Proceedings of the 1998 International Conference on Image Processing, ICIP98 (Cat. No. 98CB36269), Chicago, IL, USA.
34. Li, A., Zhao, S., Ma, X., Gong, M., Qi, J., Zhang, R., Tao, D., and Kotagiri, R. (2020, January 23–28). Short-term and long-term context aggregation network for video inpainting. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part IV 16.10.1007/978-3-030-58548-8_42 ↗
35. Wang, C., Huang, H., Han, X., and Wang, J. (2019, January 27–28). Video inpainting by jointly learning temporal structure and spatial details. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA. No. 01.10.1609/aaai.v33i01.33015232 ↗
36. Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6–12). Learning a deep convolutional network for image super-resolution. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part IV 13.10.1007/978-3-319-10593-2_13 ↗
37. Cubic splines for image interpolation and digital filtering10.1109/TASSP.1978.1163154 ↗
38. Haris, M., Shakhnarovich, G., and Ukita, N. (2018, January 18–22). Deep back-projection networks for super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00179 ↗
39. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
40. Pan, L., Dai, Y., Liu, M., and Porikli, F. (2017, January 21–26). Simultaneous stereo video deblurring and scene flow estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.739 ↗
41. Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., and Wang, O. (2017, January 21–26). Deep video deblurring for hand-held cameras. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.33 ↗
42. Adversarial spatio-temporal learning for video deblurring10.1109/TIP.2018.2867733 ↗
43. Isobe, T., Li, S., Jia, X., Yuan, S., Slabaugh, G., Xu, C., Wang, S., and Tian, Q. (2020, January 13–19). Video super-resolution with temporal group attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00803 ↗
44. Suin, M., and Rajagopalan, A.N. (2021, January 20–21). Gated spatio-temporal attention-guided video deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00771 ↗
45. Cao, J., Li, Y., Zhang, K., Liang, J., and Van Gool, L. (2021). Video Super-Resolution Transformer. arXiv.
46. Liu, D., Wen, B., Fan, Y., Loy, C.C., and Huang, T.S. (2018). Non-local recurrent network for image restoration. arXiv.
47. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
48. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv.
49. Milan, A., Leal-Taixé, L., Reid, I., Roth, S., and Schindler, K. (2016). MOT16: A benchmark for multi-object tracking. arXiv.
50. Exploiting feature and class relationships in video categorization with regularized deep neural networks10.1109/TPAMI.2017.2670560 ↗
51. Contour detection and hierarchical image segmentation10.1109/TPAMI.2010.161 ↗
52. Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (elus). arXiv.