A Lightweight Reconstruction Model via a Neural Network for a Video Super-Resolution Model
Source abstract
Abstract
Super-resolution in image and video processing has been a challenge in computer vision, with its progression creating substantial societal ramifications. More specifically, video super-resolution methodologies aim to restore spatial details while upholding the temporal coherence among frames. Nevertheless, their extensive parameter counts and high demand for computational resources challenge the deployment of existing deep convolutional neural networks on mobile platforms. In response to these concerns, our research undertakes an in-depth investigation into deep convolutional neural networks and offers a lightweight model for video super-resolution, capable of reducing computational load. In this study, we bring forward a unique lightweight model for video super-resolution, the Deep Residual Recursive Network (DRRN). The model applies residual learning to stabilize the Recurrent Neural Network (RNN) training, meanwhile adopting depth-wise separable convolution to boost the efficiency of super-resolution operations. Thorough experimental evaluations reveal that our proposed model excels in computational efficiency and in generating refined and temporally consistent results for video super-resolution. Hence, this research presents a crucial stride toward applying video super-resolution strategies on devices with resource limitations.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Image and Video Stabilization · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Academy of Broadcasting Science, No. 2, Beijing 100866, China
School of Data Science and Media Intelligence, Communication University of China, Beijing 100024, China
Funding
Academy of Broadcasting Science
References (33)
Show source-provided references
1. Deep Learning10.1038/nature14539 ↗
2. Image Super-Resolution Via Sparse Representation10.1109/TIP.2010.2050625 ↗
3. On Bayesian Adaptive Video Super Resolution10.1109/TPAMI.2013.127 ↗
4. Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., and Wu, E. (2015, January 7–12). Handling Motion Blur in Multi-Frame Super-Resolution. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
5. Dong, C., Loy, C.C., He, K., and Tang, X. (2014, January 6–12). Learning a Deep Convolutional Network for Image Super-Resolution. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10593-2_13 ↗
6. Dong, C., Loy, C.C., and Tang, X. (2016, January 11–14). Accelerating the Super-Resolution Convolutional Neural Network. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_25 ↗
7. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate Image Super-Resolution Using Very Deep Convolutional Networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
8. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
9. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
10. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., and Fu, Y. (2018, January 18–22). Residual Dense Network for Image Super-Resolution. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00262 ↗
11. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_18 ↗
12. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., and Tan, M. (2020, January 13–19). Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00545 ↗
13. Liao, R., Tao, X., Li, R., Ma, Z., and Jia, J. (2015, January 7–13). Video Super-Resolution via Deep Draft-Ensemble Learning. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.68 ↗
14. Video Super-Resolution With Convolutional Neural Networks10.1109/TCI.2016.2532323 ↗
15. Caballero, J., Ledig, C., Aitken, A., Totz, J., Wang, Z., and Shi, W. (2017, January 21–26). Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.304 ↗
16. Jo, Y., Oh, S.W., Kang, J., and Kim, S.J. (2018, January 18–22). Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00340 ↗
17. Tao, X., Gao, H., Liao, R., Wang, J., and Jia, J. (2017, January 22–29). Detail-Revealing Deep Video Super-Resolution. Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.479 ↗
18. Tian, Y., Zhang, Y., Fu, Y., and Xu, C. (2020, January 13–19). TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00342 ↗
19. Haris, M., Shakhnarovich, G., and Ukita, N. (2019, January 15–18). Recurrent Back-Projection Network for Video Super-Resolution. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00402 ↗
20. Wang, X., Chan, K.C., Yu, K., Dong, C., and Change Loy, C. (2019, January 16–17). EDVR: Video Restoration With Enhanced Deformable Convolutional Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00247 ↗
21. Chen, J., Tan, X., Shan, C., Liu, S., and Chen, Z. (2020). VESR-Net: The Winning Solution to Youku Video Enhancement and Super-Resolution Challenge. arXiv.
22. Understanding Deformable Alignment in Video Super-Resolution
23. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
24. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., and Change Loy, C. (2018, January 8–14). ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. Proceedings of the European Conference on Computer Vision Workshops, Munich, Germany.10.1007/978-3-030-11021-5_5 ↗
25. Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution10.1109/TIP.2019.2895768 ↗
26. Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation
27. A novel lightweight multi-dimension feature fusion network for single-image super-resolution reconstruction
28. Asymmetric CNN for Image Superresolution10.1109/TSMC.2021.3069265 ↗
29. Zhu, F., Jia, X., and Wang, S. (2020). Revisiting Temporal Modeling for Video Super-resolution. arXiv.
30. Chollet, F. (2017, January 21–26). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
31. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
32. Xiang, X., Tian, Y., Zhang, Y., Fu, Y., Allebach, J.P., and Xu, C. (2020, January 13–19). Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00343 ↗
33. Yi, P., Wang, Z., Jiang, K., Jiang, J., and Ma, J. (November, January 27). Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00320 ↗