Versatile Video Coding-Post Processing Feature Fusion: A Post-Processing Convolutional Neural Network with Progressive Feature Fusion for Efficient Video Enhancement
Source abstract
Abstract
Advanced video codecs such as High Efficiency Video Coding/H.265 (HEVC) and Versatile Video Coding/H.266 (VVC) are vital for streaming high-quality online video content, as they compress and transmit data efficiently. However, these codecs can occasionally degrade video quality by adding undesirable artifacts such as blockiness, blurriness, and ringing, which can detract from the viewer’s experience. To ensure a seamless and engaging video experience, it is essential to remove these artifacts, which improves viewer comfort and engagement. In this paper, we propose a deep feature fusion based convolutional neural network (CNN) architecture (VVC-PPFF) for post-processing approach to further enhance the performance of VVC. The proposed network, VVC-PPFF, harnesses the power of CNNs to enhance decoded frames, significantly improving the coding efficiency of the state-of-the-art VVC video coding standard. By combining deep features from early and later convolution layers, the network learns to extract both low-level and high-level features, resulting in more generalized outputs that adapt to different quantization parameter (QP) values. The proposed VVC-PPFF network achieves outstanding performance, with Bjøntegaard Delta Rate (BD-Rate) improvements of 5.81% and 6.98% for luma components in random access (RA) and low-delay (LD) configurations, respectively, while also boosting peak signal-to-noise ratio (PSNR).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Coding and Compression Technologies · Advanced Image Processing Techniques · Image and Video Quality Assessment
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
Department of Electronic Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
Funding
Kyung Hee University · Korea government (MSIT)
References (45)
Show source-provided references
1. Overview of the High Efficiency Video Coding (HEVC) Standard10.1109/TCSVT.2012.2221191 ↗
2. Overview of the Versatile Video Coding (VVC) Standard and its Applications10.1109/TCSVT.2021.3101953 ↗
3. Dong, C., Deng, Y., Loy, C.C., and Tang, X. (2015, January 7–13). Compression Artifacts Reduction by a Deep Convolutional Network. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.73 ↗
4. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising10.1109/TIP.2017.2662206 ↗
5. Tai, Y., Yang, J., Liu, X., and Xu, C. (2017, January 22–29). MemNet: A Persistent Memory Network for Image Restoration. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.486 ↗
6. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. (2021, January 11–17). SwinIR: Image Restoration Using Swin Transformer. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada.10.1109/ICCVW54120.2021.00210 ↗
7. Subband Adaptive Image Deblocking Using Wavelet Based Convolutional Neural Networks10.1109/ACCESS.2021.3073202 ↗
8. WCDGAN: Weakly Connected Dense Generative Adversarial Network for Artifact Removal of Highly Compressed Images10.1109/ACCESS.2021.3138106 ↗
9. MW-GAN+ for Perceptual Quality Enhancement on Compressed Video10.1109/TCSVT.2021.3128275 ↗
10. A Switchable Deep Learning Approach for In-Loop Filtering in Video Coding10.1109/TCSVT.2019.2935508 ↗
11. Partition-Aware Adaptive Switching Neural Networks for Post-Processing in HEVC10.1109/TMM.2019.2962310 ↗
12. Frame-Wise CNN-Based Filtering for Intra-Frame Quality Enhancement of HEVC Videos10.1109/TCSVT.2020.3018230 ↗
13. MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video10.1109/TPAMI.2019.2944806 ↗
14. Enhancing Quality for HEVC Compressed Videos10.1109/TCSVT.2018.2867568 ↗
15. One-for-all: An Efficient Variable Convolution Neural Network for In-loop Filter of VVC10.1109/TCSVT.2021.3089498 ↗
16. Li, Y., Zhang, L., and Zhang, K. (2021, January 19–22). Convolutional Neural Network Based In-Loop Filter for VVC Intra Coding. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.10.1109/ICIP42928.2021.9506027 ↗
17. Chen, S., Chen, Z., Wang, Y., and Liu, S. (2020, January 6–8). In-Loop Filter with Dense Residual Convolutional Neural Network for VVC. Proceedings of the 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), Shenzhen, China.10.1109/MIPR49039.2020.00038 ↗
18. Bonnineau, C., Hamidouche, W., Travers, J.-F., Sidaty, N., and Deforges, O. (2021). Multitask Learning for VVC Quality Enhancement and Super-Resolution. arXiv.10.1109/PCS50896.2021.9477492 ↗
19. Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding10.1109/TIP.2019.2896489 ↗
20. Zhang, F., Feng, C., and Bull, D.R. (2020, January 6–10). Enhancing VVC Through CNN-Based Post-Processing. Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME), London, UK.10.1109/ICME46284.2020.9102912 ↗
21. MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering10.1109/JSTSP.2020.3043064 ↗
22. Attention-Based Dual-Scale CNN In-Loop Filter for Versatile Video Coding10.1109/ACCESS.2019.2944473 ↗
23. Adaptive postfiltering of transform coefficients for the reduction of blocking artifacts10.1109/76.920189 ↗
24. Shen, S., Fang, X., and Wang, C. (2011, January 15–17). Adaptive non-local means filtering for image deblocking. Proceedings of the 2011 4th International Congress on Image and Signal Processing, Shanghai, China.10.1109/CISP.2011.6100331 ↗
25. Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images10.1109/TIP.2007.891788 ↗
26. Blocking Artifacts Suppression in Block-Coded Images Using Overcomplete Wavelet Representation10.1109/TCSVT.2004.825555 ↗
27. Liu, X., Wu, X., Zhou, J., and Zhao, D. (2015, January 7–12). Data-driven sparsity-based restoration of JPEG-compressed images in dual transform-pixel domain. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299153 ↗
28. Ren, J., Liu, J., Li, M., Bai, W., and Guo, Z. (2013, January 20–22). Image Blocking Artifacts Reduction via Patch Clustering and Low-Rank Minimization. Proceedings of the 2013 Data Compression Conference, Snowbird, UT, USA.
29. Compression Artifact Reduction by Overlapped-Block Transform Coefficient Estimation with Block Similarity10.1109/TIP.2013.2274386 ↗
30. Kathariya, B., Li, Z., Wang, H., and Van Der Auwera, G. (2022, January 13–16). Multi-stage Locally and Long-range Correlated Feature Fusion for Learned In-loop Filter in VVC. Proceedings of the 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), Suzhou, China.10.1109/VCIP56404.2022.10008834 ↗
31. Santamaria, M., Lam, Y.-H., Cricri, F., Lainema, J., Youvalari, R.G., Zhang, H., Hannuksela, M.M., Rahtu, E., and Gaubbuj, M. (December, January 29). Content-adaptive convolutional neural network post-processing filter. Proceedings of the 2021 IEEE International Symposium on Multimedia (ISM), Naple, Italy.10.1109/ISM52913.2021.00025 ↗
32. Pham, C.D.K., Fu, C., and Zhou, J. (2021, January 19–25). Deep Learning based Spatial-Temporal In-loop filtering for Versatile Video Coding. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, TN, USA.10.1109/CVPRW53098.2021.00206 ↗
33. Lim, W.-Q., Stallenberger, B., Pfaff, J., Schwarz, H., Marpe, D., and Wiegand, T. (2024, January 12–14). Simplified CNN In-Loop Filter with fixed Classifications. Proceedings of the 2024 Picture Coding Symposium (PCS), Taichung, Taiwan.10.1109/PCS60826.2024.10566438 ↗
34. Zhang, H., Jung, C., Liu, Y., and Li, M. (2023, January 8–11). Lightweight CNN-Based in-Loop Filter for VVC Intra Coding. Proceedings of the 2023 IEEE International Conference on Image Processing (ICIP), Kuala Lumpur, Malaysia.10.1109/ICIP49359.2023.10223094 ↗
35. Lim, W.Q., Pfaff, J., Stallenberger, B., Erfurt, J., Schwarz, H., Marpe, D., and Wiegand, T. (2022, January 16–19). Adaptive Loop Filter with a CNN-Based Classification. Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France. Available online: https://ieeexplore.ieee.org/document/9897666.10.1109/ICIP46576.2022.9897666 ↗
36. QA-Filter: A QP-Adaptive Convolutional Neural Network Filter for Video Coding10.1109/TIP.2022.3152627 ↗
37. Wang, H., Jia, L., Ren, H., and Jia, K. (2023, January 26–28). Video Coding Scheme Using Pre-Processing Degradation and Post-Processing Restoration. Proceedings of the 2023 2nd International Conference on Image Processing and Media Computing (ICIPMC), Xi’an, China.10.1109/ICIPMC58929.2023.00026 ↗
38. Wieckowski, A., Brandenburg, J., Hinz, T., Bartnik, C., George, V., Hege, G., Helmrich, C., Henkel, A., Lehmann, C., and Stoffers, C. (2021, January 5–9). Vvenc: An Open and Optimized VVC Encoder Implementation. Proceedings of the 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shenzhen, China.10.1109/ICMEW53276.2021.9455944 ↗
39. Wieckowski, A., Hege, G., Bartnik, C., Lehmann, C., Stoffers, C., Bross, B., and Marpe, D. (2020, January 25–28). Towards A Live Software Decoder Implementation for The Upcoming Versatile Video Coding (VVC) Codec. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP40778.2020.9191199 ↗
40. BVI-DVC: A Training Database for Deep Video Compression10.1109/TMM.2021.3108943 ↗
41. Alshina, E., Liao, R.-L., Liu, S., and Segall, A. (2023). JVET Common Test Conditions and Evaluation Procedures for Neural Network Based Video Coding Technology, Joint Video Experts Team (JVET). JVET-AC2016-v1.
42. (2023, March 22). ffmpeg Documentation. Available online: https://ffmpeg.org/ffmpeg.html.
43. Bjontegaard, G. (2001). Calculation of Average PSNR Differences between RD-Curves, Video Coding Experts Group (VCEG). VCEG-M33.
44. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, Curran Associates, Inc.. Available online: https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
45. WCDANN: A Lightweight CNN Post-Processing Filter for VVC-Based Video Compression10.1109/ACCESS.2023.3301145 ↗