Video Multi-Scale-Based End-to-End Rate Control in Deep Contextual Video Compression
Source abstract
Abstract
In recent years, video data have increased in size, which results in enormous transmission pressure. Rate control plays an important role in stabilizing video stream transmissions by balancing the rate and distortion of video compression. To achieve high-quality videos through low-bandwidth transmission, video multi-scale-based end-to-end rate control is proposed. First, to reduce video data, the original video is processed using multi-scale bicubic downsampling as the input. Then, the end-to-end rate control model is implemented. By fully using the temporal coding correlation, a two-branch residual-based network and a two-branch regression-based network are designed to obtain the optimal bit rate ratio and Lagrange multiplier λ for rate control. For restoring high-resolution videos, a hybrid efficient distillation SISR network (HEDS-Net) is designed to build low-resolution and high-resolution feature dependencies, in which a multi-branch distillation network, a lightweight attention LCA block, and an upsampling network are used to transmit deep extracted frame features, enhance feature expression, and improve image detail restoration abilities, respectively. The experimental results show that the PSNR and SSIM BD rates of the proposed multi-scale-based end-to-end rate control are −1.24% and −0.50%, respectively, with 1.82% rate control accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Image and Video Stabilization · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Urban Railway Transportation, Shanghai University of Engineering Science, Shanghai 201620, China
Funding
the major project of Zhangjiang
References (44)
Show source-provided references
1. Efficient modeling of VBR MPEG-1 coded video sources10.1109/76.825864 ↗
2. Rate control for MPEG video coding10.1016/S0923-5965(99)00009-0 ↗
3. Scalable rate control for MPEG-4 video10.1109/76.867926 ↗
4. CCXITT, and SGXV (1989). Description of Reference Model 8 (RM8), Specialists Group on Coding for Visual Telephony. Document 525.
5. Modified TMN8 rate control for low-delay video communications10.1109/TCSVT.2004.828333 ↗
6. Ma, S. (2002). Proposed Draft Description of Rate Control on JVT Standard, Joint Video Team. Doc. JVT-F086, Tech. Rep.
7. Pixel-wise unified rate-quantization model for multi-level rate control10.1109/JSTSP.2013.2272241 ↗
8. Li, B., Li, H., Li, L., and Zhang, J. (2012, January 10–19). Rate control by R-lambda model for HEVC. Proceedings of the 11th Meeting on JCTVC-K0103, JCTVC of ISO/IEC and ITU-T, Shanghai, China.
9. λ domain rate control algorithm for high efficiency video coding10.1109/TIP.2014.2336550 ↗
10. Distortion propagation-based optimal λ decision for random access rate control in HEVC10.1117/1.JEI.29.1.013002 ↗
11. Li, Y., Liu, Z., and Chen, Z. (2020, January 25–28). Rate Control for Versatile Video Coding. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP40778.2020.9191125 ↗
12. Overview on deep learning
13. Convolutional neural network-based optimal R-λ intra rate control in Versatile Video Coding10.1117/1.JEI.31.6.063011 ↗
14. A convolutional neural network-based rate control algorithm for VVC intra coding10.1016/j.displa.2024.102652 ↗
15. Neural network based rate control for versatile video coding10.1109/TCSVT.2023.3262303 ↗
16. An end-to-end compression framework based on convolutional neural networks10.1109/TCSVT.2017.2734838 ↗
17. Minnen, D., Ballé, J., and Toderici, G. (2018). Joint autoregressive and hierarchical priors for learnedimage compression. arXiv.
18. Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., and Gao, Z. (2019, January 15–20). DVC: An end-to-end deep video compression framework. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01126 ↗
19. Deep contextual video compression
20. End-to-end video compression for surveillance and conference videos10.1007/s11042-022-13484-w ↗
21. Çetin, E., Yılmaz, M.A., and Tekalp, A.M. (2022, January 16–19). Flexible-rate learned hierarchical bi-directional video compression with motion refinement and frame-level bit allocation. Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France.10.1109/ICIP46576.2022.9897455 ↗
22. Li, Y., Chen, X., Li, J., Wen, J., Han, Y., Liu, S., and Xu, X. (2022, January 23–27). Rate control for learned video compression. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746080 ↗
23. Deep Learning and Image Super-Resolution-Guided Beam and Power Allocation for mmWave Networks
24. Survey on single image based superresolution1—Implementation challenges and solutions10.1007/s11042-019-08254-0 ↗
25. Super-resolution: A comprehensive survey10.1007/s00138-014-0623-4 ↗
26. Shahar, O., Faktor, A., and Irani, M. (2011, January 20–25). Space-time super-resolution from a single video. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.10.1109/CVPR.2011.5995360 ↗
27. Deep learning for multiple-image super resolution10.1109/LGRS.2019.2940483 ↗
28. Salvetti, F., Mazzia, V., Khaliq, A., and Chiaberge, M. (2020). Multi-image super resolution of remotely sensed images using residual attention deep neural networks. Remote Sens., 12.10.3390/rs12142207 ↗
29. An end-to-end learning framework for video compression10.1109/TPAMI.2020.2988453 ↗
30. Hu, Z., Chen, Z., Xu, D., Lu, G., Ouyang, W., and Gu, S. (2020). Improving deep video compression by resolution-adaptive flow coding. European Conference on Computer Vision, Springer.10.1007/978-3-030-58536-5_12 ↗
31. Lin, J., Liu, D., Li, H., and Wu, F. (2020, January 13–19). M-LVC: Multiple frames prediction for learned video compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00360 ↗
32. Yang, R., Mentzer, F., Gool, L.V., and Timofte, R. (2020, January 13–19). Learning for video compression with hierarchical quality and recurrent enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00666 ↗
33. Yang, R., Yang, Y., Marino, J., and Mandt, S. (2021, January 3–7). Hierarchical autoregressive modeling for neural video compression. Proceedings of the 9th International Conference on Learning Representations, ICLR, Virtually.
34. Video enhancement with task-oriented flow10.1007/s11263-018-01144-2 ↗
35. BVI-DVC: A training database for deep video compression10.1109/TMM.2021.3108943 ↗
36. Agustsson, E., and Timofte, R. (2017, January 21–26). Ntire 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.150 ↗
37. Sketch-based manga retrieval using manga109 dataset10.1007/s11042-016-4020-z ↗
38. Martin, D., Fowlkes, C., Tal, D., and Malik, J. (2001, January 7–14). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings of the Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada.
39. Marco, B., Roumy, A., Guillemot, C.M., and Alberi-Morel, M.-L. (2012, January 3–7). Low-complexity single-image super-resolution based on nonnegative neighbor embedding. Proceedings of the British Machine Vision Conference, Surrey, UK.
40. Zeyde, R., Elad, M., and Protter, M. (2012, January 24–30). On single image scale-up using sparse-representations. Proceedings of the 7th International Conference of Curves and Surfaces, Avignon, France.10.1007/978-3-642-27413-8_47 ↗
41. Learning a deep convolutional network for image super-resolution reconstruction
42. Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27–30). Accurate Image Super-Resolution Using very Deep Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
43. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
44. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8–14). Image super-resolution using very deep residual channel attention networks. Proceedings of the 15th European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_18 ↗