Video Super-Resolution with Regional Focus for Recurrent Network
Source abstract
Abstract
Video super-resolution reconstruction is the process of reconstructing low-resolution video frames into high-resolution video frames. Most of the current methods use motion estimation and motion compensation to extract temporal series information, but the inaccuracy of motion estimation will lead to the degradation of the quality of video super-resolution results. Additionally, when using convolution network to extract feature information, the number of feature information is limited by the number of feature channels, resulting in poor reconstruction results. In this paper, we propose a recurrent structure of regional focus network for video super-resolution, which can avoid the influence of inaccurate motion compensation on super-resolution results. Meanwhile, regional focus blocks in the network can focus on different areas of video frames, extract different features from shallow to deep layers, and skip-connect to the last layer of the network through feature aggregation to improve the richness of features participating in the reconstruction. The experimental results show that our method has higher computational efficiency and better video super-resolution results than other temporal modeling methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, China
School of Printing, Packaging and Digital Media, Xi’an University of Technology, Xi’an 710054, China
Funding
Research and Development of Manufacturing Information System Platform Supporting Product Life Cycle Management · Natural Science Foundation of Shaanxi Province · Scientific Research Program Funded of Shaanxi Education
References (46)
Show source-provided references
1. Liu, H., Xiong, R., Qiang, S., Feng, W., and Wen, G. (2017, January 10–13). Image super-resolution based on adaptive joint distribution modeling. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), St. Petersburg, FL, USA.10.1109/VCIP.2017.8305142 ↗
2. Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., and Shi, W. (2017, January 21–26). Real-Time Video Super-Resolution With Spatio-Temporal Networks and Motion Compensation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.304 ↗
3. Video super-resolution with convolutional neural networks10.1109/TCI.2016.2532323 ↗
4. Sajjadi, M.S.M., Vemulapalli, R., and Brown, M. (2018, January 18–22). Frame-Recurrent Video Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00693 ↗
5. Tao, X., Gao, H., Liao, R., Wang, J., and Jia, J. (2017, January 22–29). Detail-revealing deep video super-resolution. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.479 ↗
6. Video enhancement with task-oriented flow10.1007/s11263-018-01144-2 ↗
7. Fuoli, D., Gu, S., and Timofte, R. (2019, January 27–28). Efficient video super-resolution through recurrent latent space propagation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW.2019.00431 ↗
8. Haris, M., Shakhnarovich, G., and Ukita, N. (2019, January 16–17). Recurrent back-projection network for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00402 ↗
9. Jo, Y., Oh, S.W., Kang, J., and Kim, S.J. (2018, January 18–23). Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00340 ↗
10. Wang, X., Chan, K.C., Yu, K., Dong, C., and Change Loy, C. (2019, January 16–17). Edvr: Video restoration with enhanced deformable convolutional networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00247 ↗
11. Yi, P., Wang, Z., Jiang, K., Jiang, J., and Ma, J. (2019, January 27–28). Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00320 ↗
12. Kim, S.Y., Lim, J., Na, T., and Kim, M. (2018). 3dsrnet: Video super-resolution using 3d convolutional neural networks. arXiv.
13. Deep video super-resolution using HR optical flow estimation10.1109/TIP.2020.2967596 ↗
14. Song, Q., and Liu, H. (2021). Deep Gradient Prior Regularized Robust Video Super-Resolution. Electronics, 10.10.3390/electronics10141641 ↗
15. Huang, Y., Wang, W., and Wang, L. (2015, January 7–12). Bidirectional recurrent convolutional networks for multi-frame super-resolution. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, USA.
16. Isobe, T., Jia, X., Gu, S., Li, S., Wang, S., and Tian, Q. (2020, January 23–28). Video super-resolution with recurrent structure-detail network. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58610-2_38 ↗
17. Isobe, T., Li, S., Jia, X., Yuan, S., Slabaugh, G., Xu, C., Li, Y.L., Wang, S., and Tian, Q. (2020, January 13–19). Video super-resolution with temporal group attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00803 ↗
18. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
19. Isobe, T., Zhu, F., Jia, X., and Wang, S. (2020). Revisiting temporal modeling for video super-resolution. arXiv.
20. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
21. Kim, T.H., Sajjadi, M.S., Hirsch, M., and Scholkopf, B. (2018, January 8–14). Spatio-temporal transformer network for video restoration. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_7 ↗
22. Xiao, Z., Fu, X., Huang, J., Cheng, Z., and Xiong, Z. (2021, January 20–25). Space-time distillation for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00215 ↗
23. Multi-memory convolutional neural network for video super-resolution10.1109/TIP.2018.2887017 ↗
24. Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement10.1109/TPAMI.2019.2941941 ↗
25. Video super-resolution using non-simultaneous fully recurrent convolutional network10.1109/TIP.2018.2877334 ↗
26. Real-time video super-resolution via motion convolution kernel estimation10.1016/j.neucom.2019.07.089 ↗
27. Kalarot, R., and Porikli, F. (2019, January 16–17). Multiboot vsr: Multi-stage multi-reference bootstrapping for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00258 ↗
28. Learning temporal coherence via self-supervision for GAN-based video generation10.1145/3386569.3392457 ↗
29. Li, W., Tao, X., Guo, T., Qi, L., Lu, J., and Jia, J. (2020, January 23–28). Mucan: Multi-correspondence aggregation network for video super-resolution. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58607-2_20 ↗
30. Li, Y., Zhu, H., Hou, Q., Wang, J., and Wu, W. (2022). Video Super-Resolution Using Multi-Scale and Non-Local Feature Fusion. Electronics, 11.10.3390/electronics11091499 ↗
31. Isobe, T., Jia, X., Tao, X., Li, C., Li, R., Shi, Y., Mu, J., Lu, H., and Tai, Y.W. (2022, January 19–20). Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01689 ↗
32. Chan, K.C., Wang, X., Yu, K., Dong, C., and Loy, C.C. (2021, January 20–25). BasicVSR: The search for essential components in video super-resolution and beyond. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00491 ↗
33. Chan, K.C., Zhou, S., Xu, X., and Loy, C.C. (2022, January 19–20). BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00588 ↗
34. Deformable 3d convolution for video super-resolution10.1109/LSP.2020.3013518 ↗
35. Chen, J., Tan, X., Shan, C., Liu, S., and Chen, Z. (2020). Vesr-net: The winning solution to youku video enhancement and super-resolution challenge. arXiv.
36. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. (2017, January 22–29). Deformable convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.89 ↗
37. Tian, Y., Zhang, Y., Fu, Y., and Xu, C. (2020, January 13–19). Tdan: Temporally-deformable alignment network for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00342 ↗
38. Yan, B., Lin, C., and Tan, W. (2019, January 8–12). Frame and feature-context video super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence, Atlanta, GA, USA.
39. Li, S., He, F., Du, B., Zhang, L., Xu, Y., and Tao, D. (2019, January 16–17). Fast spatio-temporal residual network for video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01077 ↗
40. Liu, H., Zhao, P., Ruan, Z., Shang, F., and Liu, Y. (2021, January 2–9). Large motion video super-resolution with dual subnet and multi-stage communicated upsampling. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i3.16310 ↗
41. Zhu, X., Li, Z., Zhang, X.Y., Li, C., Liu, Y., and Xue, Z. (February, January 27). Residual invertible spatio-temporal network for video super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
42. Li, Y., Zhu, H., and Yu, S. (2022). High-Magnification Super-Resolution Reconstruction of Image with Multi-Task Learning. Electronics, 11.10.3390/electronics11091412 ↗
43. Lee, J., Lee, M., Cho, S., and Lee, S. (2022, January 19–20). Reference-based Video Super-Resolution Using Multi-Camera Video Triplets. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01730 ↗
44. Anisotropic angle distribution learning for head pose estimation and attention understanding in human-computer interaction10.1016/j.neucom.2020.09.068 ↗
45. MFDNet: Collaborative poses perception and matrix Fisher distribution for head pose estimation10.1109/TMM.2021.3081873 ↗
46. On Bayesian adaptive video super resolution10.1109/TPAMI.2013.127 ↗