Survey on Quality of Experience Evaluation for Cloud-Based Interactive Applications
Source abstract
Abstract
A cloud-based interactive application (CIA) is an application running in the cloud with stringent interactivity requirements, such as remote desktop and cloud gaming. These services have experienced a surge in usage, primarily due to the adoption of new remote work practices during the pandemic and the emergence of entertainment schemes similar to cloud gaming platforms. Evaluating the quality of experience (QoE) in these applications requires specific metrics, including interactivity time, responsiveness, and the assessment of video- and audio-quality degradation. Despite existing studies that evaluate QoE and compare features of general cloud applications, systematic research into QoE for CIAs is lacking. Previous surveys often narrow their focus, overlooking a comprehensive assessment. They touch on QoE in broader contexts but fall short in detailed metric analysis. Some emphasise areas like mobile cloud computing, omitting CIA-specific nuances. This paper offers a comprehensive survey of QoE measurement techniques in CIAs, providing a taxonomy of input metrics, strategies, and evaluation architectures. State-of-the-art proposals are assessed, enabling a comparative analysis of their strengths and weaknesses and identifying future research directions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image and Video Quality Assessment · Cloud Computing and Remote Desktop Technologies · Advanced Optical Imaging Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical, Electronic and Communications Engineering, Public University of Navarre, 31006 Pamplona, Spain
Institute of Smart Cities, Calle Tajonar 22, 31006 Pamplona, Spain
Funding
Spanish State Research Agency
References (100+)
Show source-provided references
1. (2024, February 25). Markets and Markets Cloud Computing Market by Service Model (Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS)), Deployment Model (Public and Private), Organization Size, Vertical, and Region-Global Forecast to 2028. Available online: https://www.marketsandmarkets.com/Market-Reports/cloud-computing-market-234.html.
2. CloudChain: A Cloud Blockchain Using Shared Memory Consensus and RDMA
3. Enterprise Adoption of Cloud Computing with Application Portfolio Profiling and Application Portfolio Assessment10.1186/s13677-020-00210-w ↗
4. Markatchev, N., Curry, R., Kiddle, C., Mirtchovski, A., Simmonds, R., and Tan, T. (2009, January 9–11). A Cloud-Based Interactive Application Service. Proceedings of the 2009 Fifth IEEE International Conference on E-Science, Oxford, UK.10.1109/e-Science.2009.23 ↗
5. Wang, H., Shea, R., Ma, X., Wang, F., and Liu, J. (2014, January 21–24). On Design and Performance of Cloud-Based Distributed Interactive Applications. Proceedings of the 2014 IEEE 22nd International Conference on Network Protocols, Raleigh, NC, USA.10.1109/ICNP.2014.25 ↗
6. Menychtas, A., Kyriazis, D., Gogouvitis, S., Oberle, K., Voith, T., Galizo, G., Berger, S., Oliveros, E., and Boniface, M. (2011, January 7–9). A Cloud Platform for Real-Time Interactive Applications. Proceedings of the CLOSER 2011, Noordwijkerhout, The Netherlands.
7. International Telecommunication Union (2007). Recommendation P.10/G.100: Definition of Quality of Experience (QoE), International Telecommunication Union.
8. Toward Total Quality of Experience: A QoE Model in a Communication Ecosystem10.1109/MCOM.2012.6178834 ↗
9. Protocol-Agnostic Method for Monitoring Interactivity Time in Remote Desktop Services10.1007/s11042-021-10708-3 ↗
10. Latency-Driven Distribution: Infrastructure Needs of Participatory Entertainment Applications10.1109/MCOM.2005.1453430 ↗
11. Smart Collaborative Evolvement for Virtual Group Creation in Customized Industrial IoT10.1109/TNSE.2022.3203790 ↗
12. Wersync: A Web Platform for Synchronized Social Viewing Enabling Interaction and Collaboration10.1016/j.jnca.2020.102939 ↗
13. Bartik, A., Cullen, Z., Glaeser, E.L., Luca, M., and Stanton, C. (2020). What Jobs Are Being Done at Home During the COVID-19 Crisis? Evidence from Firm-Level Surveys. SSRN Electron. J., w27422.10.3386/w27422 ↗
14. Cho, J., Kim, S., Kim, N., and Kang, S. (2022). Development of a Remote Collaboration System for Interactive Communication with Building Information Model in Mixed Reality. Appl. Sci., 12.10.3390/app12178738 ↗
15. Lee, K., Shin, J., Kwon, S., Cho, C.S., and Chung, S. (2021). BIM Environment Based Virtual Desktop Infrastructure (VDI) Resource Optimization System for Small to Medium-Sized Architectural Design Firms. Appl. Sci., 11.10.3390/app11136160 ↗
16. Rodríguez Lera, F.J., Fernández González, D., Martín Rico, F., Guerrero-Higueras, Á.M., and Conde, M.Á. (2021). Measuring Students Acceptance and Usability of a Cloud Virtual Desktop Solution for a Programming Course. Appl. Sci., 11.10.3390/app11157157 ↗
17. Multi-Access Edge Computing Fundamentals, Services, Enablers and Challenges: A Complete Survey10.1016/j.jnca.2021.103308 ↗
18. Pandemic Programming: How COVID-19 Affects Software Developers and How Their Organizations Can Help10.1007/s10664-020-09875-y ↗
19. Predicting Employee Telecommuting Preferences and Job Outcomes amid COVID-19 Pandemic: A Latent Profile Analysis10.1007/s12144-021-02496-8 ↗
20. QoE Management of Multimedia Streaming Services in Future Networks: A Tutorial and Survey10.1109/COMST.2019.2958784 ↗
21. A Survey on the Computation Offloading Approaches in Mobile Edge Computing: A Machine Learning-Based Perspective10.1016/j.comnet.2020.107496 ↗
22. Laghari, A.A., Zhang, X., Shaikh, Z.A., Khan, A., Estrela, V.V., and Izadi, S. (2023). A Review on Quality of Experience (QoE) in Cloud Computing. J. Reliab. Intell. Environ., 1–15.10.1007/s40860-023-00210-y ↗
23. A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services
24. Study of Subjective and Objective Quality Assessment of Audio-Visual Signals10.1109/TIP.2020.2988148 ↗
25. A Survey on Cloud Gaming: Future of Computer Games10.1109/ACCESS.2016.2590500 ↗
26. Quality of Experience (QoE) in Cloud Gaming Models: A Review10.3233/MGS-190313 ↗
27. A Survey of Interactive Remote Rendering Systems10.1145/2719921 ↗
28. An Introduction to Online Video Game QoS and QoE Influencing Factors10.1109/COMST.2022.3177251 ↗
29. Delay-Sensitive Video Computing in the Cloud: A Survey10.1145/3264510 ↗
30. Tang, W., Nguyen, T.D., and Huh, E.N. (2014, January 6–9). A Survey Study on QoE Perspective of Mobile Cloud Computing. Proceedings of the 2014 International Conference on Information Science & Applications (ICISA), Seoul, Republic of Korea.10.1109/ICISA.2014.6847345 ↗
31. Quality of Experience in Cloud Services: Survey and Measurements10.1016/j.comnet.2014.01.008 ↗
32. International Telecommunication Union (1998). Recommendation ITU P.911, Subjective Audiovisual Quality Assessment Methods for Multimedia Application, International Telecommunication Union.
33. Methods for Objective and Subjective Evaluation of Zero-Client Computing10.1109/ACCESS.2019.2925083 ↗
34. An Analysis of Approaches for Desktop Virtualization and Challenges10.32628/CSEIT2174133 ↗
35. Liu, T., He, S., Huang, S., Tsang, D., Tang, L., Mars, J., and Wang, W. (2020, January 17–21). A Benchmarking Framework for Interactive 3D Applications in the Cloud. Proceedings of the 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Athens, Greece.10.1109/MICRO50266.2020.00076 ↗
36. Beyond First Impressions: Estimating Quality of Experience for Interactive Web Applications10.1109/ACCESS.2020.2979385 ↗
37. Towards Enabling Residential Virtual-Desktop Computing10.1109/TCC.2021.3113905 ↗
38. FastDesk: A Remote Desktop Virtualization System for Multi-Tenant10.1016/j.future.2017.07.001 ↗
39. Hsu, C.F., Huang, C.Y., and Liu, X. (2021, January 15–17). Measuring Objective Visual Quality of Real-time Communication Systems in the Wild. Proceedings of the 2021 IEEE Seventh International Conference on Multimedia Big Data (BigMM), Taichung, Taiwan.10.1109/BigMM52142.2021.00010 ↗
40. Saverimoutou, A., Mathieu, B., and Vaton, S. (2019, January 19–21). Web View: Measuring & Monitoring Representative Information on Websites. Proceedings of the 2019 22nd Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), Paris, France.10.1109/ICIN.2019.8685876 ↗
41. The Optimization of Network Performance Evaluation Method for Virtual Desktop QoE Based on SPICE10.1007/978-981-15-1301-5_12 ↗
42. Magaña, E., Sesma, I., Morató, D., and Izal, M. (2019). Remote Access Protocols for Desktop-as-a-Service Solutions. PLoS ONE, 14.10.1371/journal.pone.0207512 ↗
43. Shu, S., Nahrstedt, K., and Campbell, R. (2011, January 11–15). Distortion over Latency: Novel Metric for Measuring Interactive Performance in Remote Rendering Systems. Proceedings of the 2011 IEEE International Conference on Multimedia and Expo, Barcelona, Spain.10.1109/ICME.2011.6011864 ↗
44. Hossfeld, T., Metzger, F., and Rossi, D. (June, January 29). Speed Index: Relating the Industrial Standard for User Perceived Web Performance to Web QoE. Proceedings of the 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), Cagliari, Italy.10.1109/QoMEX.2018.8463430 ↗
45. (2024, February 25). Exoprise. CloudReady RDP Sensor. Available online: https://www.exoprise.com/2017/01/10/monitor-remote-desktop-performance/.
46. Wang, P.C., Ellis, A.I., Hart, J.C., and Hsu, C.H. (2017, January 22–23). Optimizing Next-Generation Cloud Gaming Platforms with Planar Map Streaming and Distributed Rendering. Proceedings of the 2017 15th Annual Workshop on Network and Systems Support for Games (NetGames), Taipei, Taiwan.10.1109/NetGames.2017.7991544 ↗
47. Wehner, N., Wassermann, S., Seufert, M., and Casas, P. (2020, January 2–6). Improving Web QoE Monitoring for Encrypted Network Traffic through Time Series Modeling. Proceedings of the 2nd Workshop on AI in Networks and Distributed Systems (WAIN), Milan, Italy.
48. Graff, P., Marchal, X., Cholez, T., Tuffin, S., Mathieu, B., and Festor, O. (2021, January 25–29). An Analysis of Cloud Gaming Platforms Behavior under Different Network Constraints. Proceedings of the 2021 17th International Conference on Network and Service Management (CNSM), Izmir, Turkey.10.23919/CNSM52442.2021.9615562 ↗
49. Measuring Thin-Client Performance Using Slow-Motion Benchmarking10.1145/592637.592640 ↗
50. Peñaherrera-Pulla, O.S., Baena, C., Fortes, S., Baena, E., and Barco, R. (2021). Measuring Key Quality Indicators in Cloud Gaming: Framework and Assessment over Wireless Networks. Sensors, 21.10.3390/s21041387 ↗
51. Nguyen, T., Calyam, P., and Antequera, R.B. (2015, January 11–15). Benchmarking in Virtual Desktops for End-to-End Performance Traceability. Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM), Ottawa, ON, Canada.10.1109/INM.2015.7140479 ↗
52. Varghese, D., Saxena, M., Sharma, A., and Ginetti, A.J.M.G. (2018). System and Method for Automated Testing of User Interface Software for Visual Responsiveness. (10,114,733), U.S. Patent.
53. Quality of Experience Framework for Cloud Computing (QoC)10.1109/ACCESS.2018.2865967 ↗
54. Quality of Experience (QoE)-Aware Placement of Applications in Fog Computing Environments10.1016/j.jpdc.2018.03.004 ↗
55. (2024, February 25). Video Signal Input Lag Tester. Leo Bodnar, Simulator Electronics. Available online: http://www.leobodnar.com/shop/index.php?main_page=product_info&cPath=89&products_id=212.
56. (2024, February 25). 4K HDMI Video Signal Lag Tester. Leo Bodnar, Simulator Electronics. Available online: http://www.leobodnar.com/shop/index.php?main_page=product_info&cPath=89&products_id=317.
57. Johnsen, M. (2024, February 25). How to Measure User’s Graphical Experience. Available online: https://www.brianmadden.com/geekout365/player/5714564652001.
58. (2024, February 25). LDAT (Latency Display Analysis Tool). Available online: https://www.nvidia.com/en-us/geforce/news/nvidia-reviewer-toolkit/.
59. (2024, February 25). TeamViewer Website. Available online: https://www.teamviewer.com/es/.
60. (2024, February 25). Google Remote Desktop. Available online: https://remotedesktop.google.com/?pli=1.
61. (2024, February 25). Windows Remote Desktop Client Website. Available online: https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients.
62. (2024, February 25). VMware View. Available online: https://www.vmware.com/products/horizon.html.
63. (2024, February 25). Citrix-All in One Workspace Solution for Secure Access. Available online: https://www.citrix.com/.
64. (2024, February 25). Amazon Workspaces. Available online: https://aws.amazon.com/es/workspaces/.
65. Teradici, H.P. (2024, February 25). Hybrid Work Report 2022: The Strategic Role of Digital Workspaces. Available online: https://reinvent.hp.com/hybrid-work-report-2022.
66. Teradici, H.P. (2024, February 25). Remote Work 2020 Report—The Separation of Work and Place. Available online: https://connect.teradici.com/remote-work-2020.
67. Dennison, K. (2024, February 25). Forbes-How The Flexible & Remote Work Debate Will Carry Into 2024. Available online: https://www.forbes.com/sites/karadennison/2024/01/24/how-the-flexible--remote-work-debate-will-carry-into-2024/.
68. (2024, February 25). Vena Solutions-Remote Work Statistics and Trends for 2024. Available online: https://www.venasolutions.com/blog/remote-work-statistics.
69. (2024, February 25). SunRay-Clients. Available online: https://docs.oracle.com/cd/E35310_01/E35309/html/DesktopClients.html.
70. (2024, February 25). Laplink® Website. Available online: https://web.laplink.com/.
71. (2024, February 25). Real VNC Website. Available online: https://www.realvnc.com/es/connect/download/viewer/.
72. (2024, February 25). NVIDIA GeForce Website. Available online: https://www.nvidia.com/es-es/geforce-now/.
73. (2024, February 25). Amazon Luna Website. Available online: https://luna.amazon.com/.
74. PlayStation (2024, February 25). PS Now Website. Available online: https://www.playstation.com/es-es/ps-now/.
75. (2024, February 25). Xbox Cloud Gaming. Available online: https://www.xbox.com/play.
76. (2024, February 25). Markets and Markets, Cloud Gaming Market by Offering (Infrastructure, Gaming Platform Services), Device Type (Smartphones, Tablets, Gaming Consoles, PCs & Laptops, Smart TVs, HMDs), Solution (Video Streaming, File Streaming), Gamer Type, Region–2023 to 2028. Available online: https://www.marketsandmarkets.com/Market-Reports/cloud-gaming-market-62740366.html.
77. Cloud Gaming with Foveated Video Encoding10.1145/3369110 ↗
78. Enhancing Video Encoding for Cloud Gaming Using Rendering Information10.1109/TCSVT.2015.2450175 ↗
79. Zhao, J., Wang, Y., Cao, Y., Guo, M., Huang, X., Zhang, R., Dou, X., Niu, X., Cui, Y., and Wang, J. (2021). The Fusion Strategy of 2D and 3D Information Based on Deep Learning: A Review. Remote Sens., 13.10.3390/rs13204029 ↗
80. (2024, February 25). Google Docs. Available online: https://www.google.es/intl/es/docs/about/.
81. (2024, February 25). Microsoft Office. Available online: https://www.office.com.
82. Van Steen, M., and Tanenbaum, A.S. (2017). Distributed Systems, Maarten van Steen Leiden.
83. Marinescu, D.C. (2022). Cloud Computing: Theory and Practice, Morgan Kaufmann.
84. Möller, S. (2000). Assessment and Prediction of Speech Quality in Telecommunications, Springer Science & Business Media.10.1007/978-1-4757-3117-0 ↗
85. User Satisfaction Oriented Resource Allocation for Fog Computing: A Mixed-Task Paradigm10.1109/TCOMM.2020.3008705 ↗
86. Hossfeld, T., Schatz, R., and Egger, S. (2011, January 7–9). SOS: The MOS Is Not Enough!. Proceedings of the 2011 Third International Workshop on Quality of Multimedia Experience, Mechelen, Belgium.10.1109/QoMEX.2011.6065690 ↗
87. International Telecommunication Union (2003). Mean Opinion Score (MOS) Terminology, ITU-T Recommendation P. 800, International Telecommunication Union.
88. Hosfeld, T., Heegaard, P.E., Varela, M., Skorin-Kapov, L., and Fiedler, M. (July, January 29). From QoS Distributions to QoE Distributions: A System’s Perspective. Proceedings of the 2020 6th IEEE Conference on Network Softwarization (NetSoft), Ghent, Belgium.10.1109/NetSoft48620.2020.9165426 ↗
89. Quality of Service (QoS): Measurements of Image Formats in Social Cloud Computing10.1007/s11042-020-09959-3 ↗
90. Image Quality Assessment: From Error Visibility to Structural Similarity10.1109/TIP.2003.819861 ↗
91. PSNR vs SSIM: Imperceptibility Quality Assessment for Image Steganography10.1007/s11042-020-10035-z ↗
92. (2024, February 25). J.247: Objective Perceptual Multimedia Video Quality Measurement in the Presence of a Full Reference. Available online: https://www.itu.int/rec/T-REC-J.247/en.
93. (2024, February 25). P.863: Perceptual Objective Listening Quality Prediction. Available online: https://www.itu.int/rec/T-REC-P.863.
94. Blind Adaptive Mask to Improve Intelligibility of Non-Stationary Noisy Speech10.1109/LSP.2021.3086405 ↗
95. An Efficient Audio Watermarking Scheme with Scrambled Medical Images for Secure Medical Internet of Things Systems10.1007/s11042-023-14357-6 ↗
96. Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence10.1109/TASLP.2021.3069302 ↗
97. (2024, February 25). TurboVNC|Main/TurboVNC. Available online: https://turbovnc.org.
98. Woo, M., Neider, J., Davis, T., and Shreiner, D. (1999). OpenGL Programming Guide: The Official Guide to Learning OpenGL, Addison-Wesley Longman Publishing Co., Inc.. Available online: https://www.opengl.org/.
99. The Influence of Human Factors on 360∘ Mulsemedia QoE10.1016/j.ijhcs.2020.102550 ↗
100. IoT Application Placement Algorithm Based on Multi-Dimensional QoE Prioritization Model in Fog Computing Environment10.1109/ACCESS.2020.3003249 ↗