Rethinking Usability in Serious Games: Designing an Instrument That Evaluates What Really Matters in Learning Contexts
Source abstract
Abstract
Serious games are increasingly recognized as valuable tools in educational settings due to their capacity to enhance learning through engaging and interactive experiences. However, existing usability evaluation instruments often fail to account for the specific features and pedagogical goals inherent to serious games, thereby limiting their utility in iterative design improvement. This study presents a usability evaluation questionnaire SGU specifically designed for serious games, developed through a systematic literature review (SLR) following the methodology of Kitchenham and Charters. The resulting tool is a standardized questionnaire intended to comprehensively assess key usability dimensions, including gameplay mechanics, interface design, narrative engagement, and gamification elements. The questionnaire was applied in multiple serious game case studies developed within educational contexts, allowing for the identification of recurring usability issues and context-specific design challenges. The findings highlight the instrument’s relevance, clarity, and diagnostic potential, demonstrating its practical value for evaluating and refining serious games. This research contributes to the field by addressing a critical gap in usability assessment and offering a structured, context-sensitive approach for improving user experience in educational game design.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Educational Games and Gamification · Persona Design and Applications · Virtual Reality Applications and Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Departamento de Informática y Ciencias de la Computación, Escuela Politécnica Nacional, Quito 170525, Ecuador
Centro de Microcomputación y Sistemas Distribuidos (CEMISID), Universidad de Los Andes, Merida 5101, Venezuela
Intelligent and Interactive Systems Laboratory, Universidad de Las Américas, Quito 170503, Ecuador
Funding
Universidad de las Américas
References (100+)
Show source-provided references
1. Santórum, M., Morales-Martínez, D., Caiza, J., Carrión-Toro, M., Galindo, J., and Acosta-Vargas, P. (2024, January 17–19). Serious Games, Serious Impact: Enhancing Ethical Education Through Interactive Learning. Proceedings of the 2024 International Conference on Computer, Information and Telecommunication Systems (CITS), Girona, Spain.10.1109/CITS61189.2024.10608026 ↗
2. Krajčovič, M., Gabajová, G., Matys, M., Grznár, P., Dulina, L., and Kohár, R. (2021). 3D Interactive Learning Environment as a Tool for Knowledge Transfer and Retention. Sustainability, 13.10.3390/su13147916 ↗
3. Cook-Chennault, K., Villanueva Alarcón, I., and Jacob, G. (2022). Usefulness of Digital Serious Games in Engineering for Diverse Undergraduate Students. Educ. Sci., 12.10.3390/educsci12010027 ↗
4. A virtual reality role-playing serious game for experiential learning10.1080/10494820.2019.1703008 ↗
5. Serious games in future skills development: A systematic review of the design approaches10.1002/cae.22557 ↗
6. Djaouti, D., Alvarez, J., and Jessel, J.P. (2011). Classifying serious games: The G/P/S model. Handbook of Research on Improving Learning and Motivation Through Educational Games, IGI Global.10.1007/978-1-4471-2161-9_3 ↗
7. Vásquez, L.M.L. (2017). Juegos Con Propósitos Educativos Como Herramientas Para la Enseñanza de la Ética. [Ph.D. Thesis, Universidad Nacional de Colombia].
8. The use and impact of game-based learning on the learning experience and knowledge retention of nursing undergraduate students: A systematic literature review10.1016/j.nedt.2022.105484 ↗
9. Virtual reality and gamification in education: A systematic review10.1007/s11423-024-10351-3 ↗
10. Aster, A., Hütt, C., Morton, C., Flitton, M., Laupichler, M.C., and Raupach, T. (2024). Development and evaluation of an emergency department serious game for undergraduate medical students. BMC Med. Educ., 24.10.1186/s12909-024-06056-z ↗
11. Immersive virtual-reality computer-assembly serious game to enhance autonomous learning10.1007/s10055-021-00607-1 ↗
12. ORUN-VR2: A VR serious game on the projectile kinematics: Design, evaluation, and learning outcomes10.1007/s10055-023-00824-w ↗
13. Pruebas de la usabilidad en usuarios discapacitados
14. Usability testing for serious games: Making informed design decisions with user data
15. E-Guess: Evaluación de usabilidad para juegos educativos
16. Kitchenham, B., and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering, Keele University. Technical Report EBSE-2007-01.
17. Damaševičius, R., Maskeliūnas, R., and Blažauskas, T. (2023). Serious Games and Gamification in Healthcare: A Meta-Review. Information, 14.10.3390/info14020105 ↗
18. Natucci, G.C., and Borges, M.A.F. (2021, January 17–20). The Experience, Dynamics and Artifacts Framework: Towards a Holistic Model for Designing Serious and Entertainment Games. Proceedings of the 2021 IEEE Conference on Games (CoG), Copenhagen, Denmark.10.1109/CoG52621.2021.9619144 ↗
19. Usability Evaluation of Educational Games: An Analysis of Culture as a Factor Affecting Children’s Educational Attainment
20. A Software Engineering Framework for Reusable Design of Personalized Serious Games for Health: Development Study10.2196/40054 ↗
21. Frasca, G. (2001). Videogames of the Oppressed: Videogames as a Means for Critical Thinking and Debate. [Master’s Thesis, Georgia Institute of Technology].
22. Videojuegos y alfabetización digital
23. Schell, J. (2008). The Art of Game Design: A Book of Lenses, CRC Press.
24. Kars, L. (2025, May 20). What Are Serious Games?. Available online: https://grendelgames.com/what-are-serious-games/#:~:text=Serious%20games%20are%20games%20with,and%20other%20businesses%20and%20industries.
25. Abt, C.C. (1987). Serious Games, University Press of America.
26. Alvarez, J., and Michaud, L. (2008). Serious Games: Advergaming, Edugaming, Training and More, IDATE.
27. Ferdig, R., and Winn, B. (2009). The Design, Play, and Experience Framework. The Design, Play, and Experience Framework, IGI Global.
28. Carrion, M., Santórum, M., Flores, H., Aguilar, J., and Perez, M. (2019, January 13–15). Serious Game, gamified applications, educational software: A comparative study. Proceedings of the 2019 International Conference on Information Systems and Software Technologies (ICI2ST), Quito, Ecuador.10.1109/ICI2ST.2019.00015 ↗
29. Carrión Toro, M.d.C. (2022). iPlus Una Metodología Centrada en el Usuario Para el Diseño de Juegos Serios. [Ph.D. Thesis, Escuela Politécnica Nacional].
30. Deterding, S., Sicart, M., Nacke, L., O’Hara, K., and Dixon, D. (2011, January 7–12). Gamification. using game-design elements in non-gaming contexts. In Proceedings of the CHI ’11 Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA. CHI EA ′11.10.1145/1979742.1979575 ↗
31. Serious games in science education: A systematic literature10.1016/j.vrih.2022.02.001 ↗
32. The Effect of Educational Games on Learning Outcomes, Student Motivation, Engagement and Satisfaction10.1177/0735633120969214 ↗
33. Didácticos, N. (2025, May 20). ¿Qué son Los Juegos Educativos?. Available online: https://www.noedidacticos.com/blog/que-son-los-juegos-educativos-0de1c230067b.
34. Hodhod, R., Hardage, H., Abbas, S., and Aldakheel, E.A. (2023). CyberHero: An Adaptive Serious Game to Promote Cybersecurity Awareness. Electronics, 12.10.3390/electronics12173544 ↗
35. Serious games to assess university students’ soft skills: Investigating the effectiveness of a gamified assessment prototype10.1080/10494820.2023.2253849 ↗
36. Serious games for environmental education10.1002/inc3.18 ↗
37. Educational games promote the development of students’ computational thinking: A meta-analytic review10.1080/10494820.2021.1931891 ↗
38. Effect of Game Teaching Assisted by Deep Reinforcement Learning on Children’s Physical Health and Cognitive Ability
39. Gros Salvat, B. (2009). Certezas e interrogantes acerca del uso de los videojuegos para el aprendizaje. Comunicación. Revista Internacional de Comunicación Audiovisual, Editorial Universidad de Sevilla. Number 7.
40. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018). Available online: https://www.iso.org/standard/63500.html.
41. La usabilidad en Ingeniería de Software: Definición y características
42. IEEE Xplore (2025, July 15). Available online: https://ieeexplore.ieee.org/Xplore/home.jsp.
43. ACM Digital Library (2025, July 15). Available online: https://www.acm.org/.
44. Elsevier (2025, July 15). Available online: https://www.elsevier.com/.
45. Google Scholar (2025, July 15). Available online: https://scholar.google.com/.
46. Springer Nature Link (2025, July 15). Available online: https://link.springer.com/.
47. A full systematic review was completed in 2 weeks using automation tools: A case study10.1016/j.jclinepi.2020.01.008 ↗
48. Zooniverse (2025, August 29). Galaxy. Available online: https://www.zooniverse.org/projects/zookeeper/galaxy-zoo/.
49. Glitchers Ltd. (2025, August 29). Sea Hero Quest. Available online: https://seaheroquest.com/.
50. Video game unleashes millions of citizen scientists on microbiome research10.1038/s41587-024-02203-5 ↗
51. Tolentino, G.P., Battaglini, C., Pereira, A.C.V., Oliveria, R.J.d., and Paula, M.G.M.d. (2011, January 4–6). Usability of Serious Games for Health. Proceedings of the 2011 Third International Conference on Games and Virtual Worlds for Serious Applications, Athens, Greece.10.1109/VS-GAMES.2011.33 ↗
52. Diah, N.M., Ismail, M., Ahmad, S., and Dahari, M.K.M. (2010, January 17–18). Usability testing for educational computer game using observation method. Proceedings of the 2010 International Conference on Information Retrieval & Knowledge Management (CAMP), Shah Alam, Malaysia.10.1109/INFRKM.2010.5466926 ↗
53. Falcão, I., da S. Souza, D., Araujo, F.P., dos S. Fernandes, J.G., Pires, Y.P., Cardoso, D.L., and Seruffo, M.C. (2019, January 5–7). Usability and Cognitive Benefits of a Serious Game to Combat Aedes Aegypti Mosquito. Proceedings of the 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS), Cordoba, Spain.10.1109/CBMS.2019.00145 ↗
54. Omar, H., and Jaafar, A. (2008, January 26–28). Playability Heuristics Evaluation (PHE) approach for Malaysian educational games. Proceedings of the 2008 International Symposium on Information Technology, Kuala Lumpur, Malaysia.10.1109/ITSIM.2008.4632053 ↗
55. Hussain, A., Mutalib, N.A., and Zaino, A. (2014, January 17–18). A usability testing on JFakih Learning Games for hearing impairment children. Proceedings of the The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M), Kuching, Malaysia.10.1109/ICT4M.2014.7020629 ↗
56. Ismail, M., Diah, N.M., Ahmad, S., Kamal, N.A.M., and Dahari, M.K.M. (2011, January 6–7). Measuring usability of educational computer games based on the user success rate. Proceedings of the 2011 International Symposium on Humanities, Science and Engineering Research, Kuala Lumpur, Malaysia.10.1109/SHUSER.2011.6008500 ↗
57. Dagdag, R.J.E., Mariano, R.V., and Atienza, R.O. (2011, January 21–24). Teacher tool for visualization and management of a technology-enhanced learning environment. Proceedings of the TENCON 2011—2011 IEEE Region 10 Conference, Bali, Indonesia.10.1109/TENCON.2011.6129045 ↗
58. Methods for Game User Research: Studying Player Behavior to Enhance Game Design10.1109/MCG.2013.61 ↗
59. Papaloukas, S., Patriarcheas, K., and Xenos, M. (2009, January 10–12). Usability Assessment Heuristics in New Genre Videogames. Proceedings of the 2009 13th Panhellenic Conference on Informatics, Corfu, Greece.10.1109/PCI.2009.14 ↗
60. Using peer tutoring in evaluating the usability of a physically interactive computer game with children10.1016/S0953-5438(03)00008-0 ↗
61. Rosyidah, U., Haryanto, H., and Kardianawati, A. (2019, January 21–22). Usability Evaluation Using GOMS Model for Education Game “Play and Learn English”. Proceedings of the 2019 International Seminar on Application for Technology of Information and Communication (iSemantic), Semarang, Indonesia.10.1109/ISEMANTIC.2019.8884268 ↗
62. Soomro, S., Ahmad, W.F.W., and Sulaiman, S. (2014, January 3–5). Evaluation of mobile games with playability heuristic evaluation system. Proceedings of the 2014 International Conference on Computer and Information Sciences (ICCOINS), Kuala Lumpur, Malaysia.10.1109/ICCOINS.2014.6868403 ↗
63. YouTube Gamers and Think-Aloud Protocols: Introducing Usability Testing10.1109/TPC.2018.2867130 ↗
64. Liao, Y.H., and Shen, C.Y. (2012, January 27–30). Heuristic Evaluation of Digital Game Based Learning: A Case Study. Proceedings of the 2012 IEEE Fourth International Conference on Digital Game and Intelligent Toy Enhanced Learning, Takamatsu, Japan.10.1109/DIGITEL.2012.54 ↗
65. Jegers, K. (2008, January 8–13). Investigating the Applicability of Usability and Playability Heuristics for Evaluation of Pervasive Games. Proceedings of the 2008 Third International Conference on Internet and Web Applications and Services, Athens, Greece.10.1109/ICIW.2008.54 ↗
66. Hookham, G., Nesbitt, K., and Kay-Lambkin, F. (2016, January 1–5). Comparing usability and engagement between a serious game and a traditional online program. Proceedings of the Australasian Computer Science Week Multiconference, Canberra, Australia. ACSW ′16.10.1145/2843043.2843365 ↗
67. Baauw, E., Bekker, M.M., and Markopoulos, P. (2006, January 7–9). Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group. Proceedings of the 2006 Conference on Interaction Design and Children, Tampere, Finland. IDC ′06.10.1145/1139073.1139095 ↗
68. Yue, W.S., and Zin, N.A.M. (2009, January 24–26). Usability evaluation for history educational games. Proceedings of the 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human, Seoul, Republic of Korea. ICIS ′09.10.1145/1655925.1656110 ↗
69. Papaloukas, S., and Xenos, M. (2008, January 16–18). Usability and education of games through combined assessment methods. Proceedings of the 1st International Conference on PErvasive Technologies Related to Assistive Environments, Athens, Greece. PETRA ′08.10.1145/1389586.1389670 ↗
70. Pinelle, D., Wong, N., and Stach, T. (2008, January 5–10). Heuristic evaluation for games: Usability principles for video game design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. CHI ′08.10.1145/1357054.1357282 ↗
71. Rodio, F., and Bastien, J.M.C. (2013, January 12–15). Heuristics for Video Games Evaluation: How Players Rate Their Relevance for Different Game Genres According to Their Experience. Proceedings of the 25th Conference on l’Interaction Homme-Machine, Talence, France. IHM ′13.10.1145/2534903.2534915 ↗
72. Pinelle, D., Wong, N., Stach, T., and Gutwin, C. (2009, January 10–13). Usability heuristics for networked multiplayer games. Proceedings of the 2009 ACM International Conference on Supporting Group Work, Sanibel Island, FL, USA. GROUP ′09.10.1145/1531674.1531700 ↗
73. Desurvire, H., Caplan, M., and Toth, J.A. (2004, January 24–29). Using heuristics to evaluate the playability of games. Proceedings of the CHI ’04 Extended Abstracts on Human Factors in Computing Systems, Vienna, Austria. CHI EA ′04.10.1145/985921.986102 ↗
74. Korhonen, H., and Koivisto, E.M.I. (2006, January 12–15). Playability heuristics for mobile games. Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services, Helsinki, Finland. MobileHCI ′06.10.1145/1152215.1152218 ↗
75. Fabiano, F., Sanchez-Francisco, M., Díaz, P., and Aedo, I. (2018, January 3–6). Evaluating a pervasive game for urban awareness. Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, Barcelona, Spain. MobileHCI ′18.10.1145/3236112.3236140 ↗
76. Evaluating the LEGO–RoboLab interface with experts10.1145/1129006.1129017 ↗
77. Mohd, F., Noor, M., Daud, E.H.C., and Hasbullah, S.S. (2016, January 4–6). Mobile Games Usability Using the Idea of Mark Overmars. Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication, Danang, Vietnam. IMCOM ′16.10.1145/2857546.2857578 ↗
78. Virtual gaming simulation of a mental health assessment: A usability study10.1016/j.nepr.2018.05.007 ↗
79. Teaching clinical reasoning and decision-making skills to nursing students: Design, development, and usability evaluation of a serious game10.1016/j.ijmedinf.2016.06.014 ↗
80. Design and implementation of a serious game on neurorehabilitation: Data on modifications of functionalities along implementation releases10.1016/j.dib.2018.08.100 ↗
81. Engaged in learning neurorehabilitation: Development and validation of a serious game with user-centered design10.1016/j.compedu.2018.06.001 ↗
82. Evaluation of simulation games for teaching engineering and manufacturing10.1016/j.procs.2012.10.073 ↗
83. A review of heuristics evaluation component for mobile educational games10.1016/j.procs.2019.11.213 ↗
84. Luna-Perejon, F., Malwade, S., Styliadis, C., Civit, J., Cascado-Caballero, D., Konstantinidis, E., Abdul, S.S., Bamidis, P.D., Civit, A., and Li, Y.C.J. (2019). Evaluation of user satisfaction and usability of a mobile app for smoking cessation. Comput. Methods Programs Biomed., 182.10.1016/j.cmpb.2019.105042 ↗
85. Amaro, B.A., and Mira, E.E. (2025, September 13). Desarrollo y Evaluación de Usabilidad de un Juego "serio” Utilizado en la Enseñanza del Comportamiento de la Bolsa de Valores y Cereales.2020. Trabajo Académico. Available online: https://ridaa.unicen.edu.ar/items/6b6cea48-b205-4830-a805-4fb313ded06a/full.
86. Usabilidad de juegos educativos
87. Metodología de Evaluación de Videojuegos Serios en Competencias Ciudadanas, Exposición Caso Civia
88. Gordon, N., Brayshaw, M., and Aljaber, T. (2016, January 17–22). Heuristic evaluation for serious immersive games and M-instruction. Proceedings of the International Conference on Learning and Collaboration Technologies, Toronto, ON, Canada.10.1007/978-3-319-39483-1_29 ↗
89. Olimpio, A., and Notargiacomo, P. (2020, January 5–7). Heuristic Usability Evaluation Applied to Educational Games. Proceedings of the International Conference on Educational Technologies 2020 (ICEduTech 2020), São Paulo, Brazil.10.33965/icedutech2020_202002L004 ↗
90. Heuristic evaluation on usability of educational games: A systematic review10.15388/infedu.2019.20 ↗
91. Isleyen, F., Gulkesen, K.H., Cinemre, B., Samur, M.K., Zayim, N., and Sen Kaya, S. (2014). Evaluation of the usability of a serious game aiming to teach facial expressions to schizophrenic patients. e-Health–For Continuity of Care, IOS Press.10.3233/978-1-61499-432-9-662 ↗
92. Co-design with Children: Using Participatory Design for Design Thinking and Social and Emotional Learning10.1515/edu-2019-0021 ↗
93. Wang, J. (2016). A Usability Assessment for a Career Planning Educational Video Game. [Master’s Thesis, Purdue University].
94. Lessons from an educational game usability evaluation10.3991/ijim.v6i2.1888 ↗
95. Mohamed, H., and Jaafar, A. (December, January 30). Development and potential analysis of heuristic evaluation for educational computer game (PHEG). Proceedings of the 5th International Conference on Computer Sciences and Convergence Information Technology, Seoul, Republic of Korea.
96. Barbosa, M.B., Rego, A.B., and De Medeiros, I. (2015, January 11–13). HEEG: Heuristic evaluation for educational games. Proceedings of the XIV Brazilian Symposium of Games and Digital Entertainment, Teresina, PI, Brazil.
97. Gómez, R.Y. (2019). Evaluación Heurística de la Usabilidad en Contextos Específicos. [Ph.D. Thesis, Universidad de Sevilla].
98. Chacón Araya, J.M. (2017). Evaluación de Usabilidad de Juegos de Video Para Casinos en Plataformas Móviles: Conceptos y Experiencias. [Ph.D. Thesis, Universidad Nacional de Costa Rica].
99. Castillo Aguirre, G. (2020). Métodos de Evaluación de la Usabilidad en Aplicaciones Móviles. [Master’s Thesis, Universidad Señor de Sipán].
100. Chanchí, G.E.G., Álvarez, M.C.G., and Campo, W.Y.M. (2020). Criterios de usabilidad para el diseño e implementación de videojuegos. Revista Ibérica de Sistemas e Tecnologias de Informação, Universidad de Medellin.