Assessment of the Quality of Video Sequences Performed by Viewers at Home and in the Laboratory
Source abstract
Abstract
The paper presents the results of subjective and objective quality assessments of H.264-, H.265-, and VP9-encoded video. Most of the literature is devoted to subjective quality assessment in well-defined laboratory circumstances. However, the end users usually watch the films in their home environments, which may be different from the conditions recommended for laboratory measurements. This may cause significant differences in the quality assessment scores. Thus, the aim of the research is to show the impact of environmental conditions on the video quality perceived by the user. The subjective assessment was made in two different environments: in the laboratory and in users’ homes, where people often watch movies on their laptops. The video signal was assessed by young viewers who were not experts in the field of quality assessment. The tests were performed taking into account different image resolutions and different bit rates. The research showed strong correlations between the obtained results and the coding bit rates used, and revealed a significant difference between the quality scores obtained in the laboratory and at home. As a conclusion, it must be underlined that the laboratory tests are necessary for comparative purposes, while the assessment of the video quality experienced by end users should be performed under circumstances that are as close as possible to the user’s home environment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image and Video Quality Assessment · Video Coding and Compression Technologies · Advanced Data Compression Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Telecommunications and Teleinformatics, Faculty of Information and Telecommunication Technology, Wrocław University of Science and Technology, Wybrzeże Wyspiańskiego 27, 50-370 Wrocław, Poland
Photonics and Microsystems Department of Acoustics, Multimedia and Signal Processing, Faculty of Electronic, Wrocław University of Science and Technology, Wybrzeże Wyspiańskiego 27, 50-370 Wrocław, Poland
References (49)
Show source-provided references
1. (2020). Methodologies for the Subjective Assessment of the Quality of Television Images (Standard No. ITU-R BT 500-14).
2. Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality10.17743/jaes.2022.0063 ↗
3. QoE Modeling for HTTP Adaptive Video Streaming–A Survey and Open Challenges10.1109/ACCESS.2019.2901778 ↗
4. Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures10.1109/MSP.2008.930649 ↗
5. The accuracy of PSNR in predicting video quality for different video scenes and frame rates10.1007/s11235-010-9351-x ↗
6. Klink, J., and Uhl, T. (2020, January 17–19). Video Quality Assessment: Some Remarks on Selected Objective Metrics. Proceedings of the 2020 28th International Conference Software, Telecommun. Comput. Networks, SoftCOM, Split, Croatia.10.23919/SoftCOM50211.2020.9238303 ↗
7. Vranjes, M., Rimac-Drlje, S., and Zagar, D. (200, January 12–14). Objective video quality metrics. Proceedings of the ELMAR 2007, Zadar, Croatia.10.1109/ELMAR.2007.4418797 ↗
8. Kotevski, Z., and Mitrevski, P. (2010, January 23–26). Performance Assessment of Metrics for Video Quality Estimation. Proceedings of the International Scientific Conference on Information, Communication and Energy Systems and Technologies, Macedonia, Greece.
9. Objective video quality assessment methods: A classification, review, and performance comparison10.1109/TBC.2011.2104671 ↗
10. Akramullah, S., and Akramullah, S. (2014). Digital Video Concepts, Methods, and Metrics, Apress.10.1007/978-1-4302-6713-3 ↗
11. From QoS to QoE: A Tutorial on Video Quality Assessment10.1109/COMST.2014.2363139 ↗
12. Hanhart, P., Korshunov, P., and Ebrahimi, T. (2013, January 1–3). Benchmarking of quality metrics on ultra-high definition video sequences. Proceedings of the 2013 18th International Conference on Digital Signal Processing (DSP), Santorini, Greece.10.1109/ICDSP.2013.6622760 ↗
13. Benchmarking of objective quality metrics for HDR image quality assessment10.1186/s13640-015-0091-4 ↗
14. Klink, J. (2021). A Method of Codec Comparison and Selection for Good Quality Video Transmission Over Limited-Bandwidth Networks. Sensors, 21.10.3390/s21134589 ↗
15. Barman, N., and Martini, M.G. (June, January 31). H. 264/MPEG-AVC, H. 265/MPEG-HEVC and VP9 codec comparison for live gaming video streaming. Proceedings of the 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany.10.1109/QoMEX.2017.7965686 ↗
16. Perceptual-based quality assessment for audio–visual services: A survey10.1016/j.image.2010.02.002 ↗
17. Why is Multimedia Quality of Experience Assessment a Challenging Problem?10.1109/ACCESS.2019.2936470 ↗
18. Rassool, R. (2017, January 7–9). VMAF reproducibility: Validating a perceptual practical video quality metric. Proceedings of the 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Cagliari, Italy.10.1109/BMSB.2017.7986143 ↗
19. VQAMap: A Novel Mechanism for Mapping Objective Video Quality Metrics to Subjective MOS Scale10.1109/TBC.2016.2570002 ↗
20. A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP10.1109/COMST.2018.2862938 ↗
21. Adaptive Bitrate Selection: A Survey10.1109/COMST.2017.2725241 ↗
22. Zabrovskiy, A., Feldmann, C., and Timmerer, C. (2018, January 12–15). Multi-codec DASH dataset. Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam, The Netherlands.10.1145/3204949.3208140 ↗
23. Visual-PSNR measure of image quality10.1016/j.jvcir.2014.01.008 ↗
24. Image Quality Assessment: From Error Visibility to Structural Similarity10.1109/TIP.2003.819861 ↗
25. Spatiotemporal Feature Integration and Model Fusion for Full Reference Video Quality Assessment10.1109/TCSVT.2018.2868262 ↗
26. A universal image quality index10.1109/97.995823 ↗
27. No-Reference Image Blur Assessment Based on Discrete Orthogonal Moments10.1109/TCYB.2015.2392129 ↗
28. Referenceless Measure of Blocking Artifacts by Tchebichef Kernel Analysis10.1109/LSP.2013.2294333 ↗
29. A No-Reference Metric for Perceived Ringing Artifacts in Images10.1109/TCSVT.2009.2035848 ↗
30. Zhu, H., Li, L., Wu, J., Dong, W., and Shi, G. (2020–19, January 14). MetaIQA: Deep meta-learning for no-reference image quality assessment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01415 ↗
31. Kang, L., Ye, P., Li, Y., and Doermann, D. (2014, January 23–28). Convolutional neural networks for no-reference image quality assessment. Proceedings of the 2014 IEEE Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.224 ↗
32. Deep neural networks for no-reference and full-reference image quality assessment10.1109/TIP.2017.2760518 ↗
33. Blind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network10.1109/TCSVT.2018.2886771 ↗
34. GraphIQA: Learning Distortion Graph Representations for Blind Image Quality Assessment
35. LIQA: Lifelong Blind Image Quality Assessment10.1109/TMM.2022.3197364 ↗
36. (2021). H.264. Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services-Coding of Moving Video, Advanced Video Coding for Generic Audiovisual Services (Standard No. ITU-T Rec).
37. (2021). H.265. Infrastructure of Audiovisual Services—Coding of Moving Video. High Efficiency Video Coding (Standard No. ITU-T Rec).
38. Grange, A., De Rivaz, P., and Hunt, J. (2023, February 25). VP9 Bitstream Decoding Process Specification. Available online: http://downloads.webmproject.org.storage.googleapis.com/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf.
39. (2021). P.910. Subjective Video Quality Assessment Methods for Multimedia Applications (Standard No. ITU-T Rec).
40. Mukherjee, D., Bankoski, J., Grange, A., Han, J., Koleszar, J., Wilkins, P., Xu, Y., and Bultje, R. (2013, January 8–11). The latest open-source video codec VP9—An overview and preliminary results. Proceedings of the 2013 Picture Coding Symposium, San Jose, CA, USA.10.1109/PCS.2013.6737765 ↗
41. Winkler, S. (2009, January 8–10). Video quality measurement standards—Current status and trends. Proceedings of the 2009 7th International Conference on Information, Communications and Signal Processing (ICICS), Macau, China.10.1109/ICICS.2009.5397585 ↗
42. Winkler, S. (2022, January 5–7). On the properties of subjective ratings in video quality experiments. Proceedings of the 2009 International Workshop on Quality of Multimedia Experience, Lippstadt, Germany.10.1109/QOMEX.2009.5246961 ↗
43. (2021). Methods for the Subjective Assessment of Video Quality, Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in Any Environment (Standard No. ITU-T Recommendation P.913).
44. International Telecommunication Union-Radiocommunication Sector P. 837-6 and P. 837-7 performance to estimate Indonesian rainfall10.12928/telkomnika.v18i5.14316 ↗
45. (2015). Parameter Values for the HDTV Standards for Production and International Programme Exchange BT Series Broadcasting Service (Standard No. ITU-R BT 709-6).
46. An automated model for the assessment of QoE of adaptive video streaming over wireless networks10.1007/s11042-021-10934-9 ↗
47. Mercat, A., Viitanen, M., and Vanne, J. (June, January 8–). UVG dataset: 50/120fps 4K sequences for video codec analysis and development. Proceedings of the 11th ACM Multimedia Systems Conference, Istanbul, Turkey.
48. (2023, February 25). FFmpeg: A Complete, Cross -Platform Solution to Record, Convert and Stream Audio and Video. Available online: https://ffmpeg.org/.
49. Brachmański, S., and Klink, J. (2022, January 22–24). Subjective Assessment of the Quality of Video Sequences by the Young Viewers. Proceedings of the 30 th International Conference on Software, Telecommunications and Computer Networks (SoftCOM 2022), Split, Croatia.10.23919/SoftCOM55329.2022.9911375 ↗