The Impact of a Digital Game-Based AI Chatbot on Students’ Academic Performance, Higher-Order Thinking, and Behavioral Patterns in an Information Technology Curriculum
Source abstract
Abstract
In the age of intelligence, information technology (IT) education has become the focus of attention in the education sector. However, traditional teaching methods fall short in motivating students and fostering higher-order thinking and have difficulty providing a personalized learning experience. Although AI chatbots can provide instant feedback as an innovative teaching tool, it is still challenging to fully enhance learner engagement. Based on this, this study developed a digital game-based AI chatbot system to enhance students’ learning experience through digital game-based learning strategies. This study utilized a quasi-experimental design with the experimental group using a digital game-based AI chatbot and the control group using a traditional AI chatbot. A comparison was made between the two groups concerning student learning performance in IT courses, higher-order thinking (including problem-solving, computational thinking, and creativity), learning motivation, and flow experience. In addition, the behavioral patterns of high-achieving and low-achieving students in the experimental group were analyzed. The results showed that the experimental group was significantly better than the control group in academic performance, problem-solving, computational thinking, learning motivation, and flow experience, but there was no significant difference in creativity tendency. Behavioral pattern analysis showed that high-achieving students in the experimental group showed more systematic learning strategies, while low-achieving students relied more on immediate feedback and external help, but both high- and low-achieving groups were able to actively talk to the AI chatbot and actively explore problem-solving strategies in the digital game. Therefore, AI chatbots based on digital games can be effectively used in IT courses to help students construct knowledge and develop higher-order thinking.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Educational Games and Gamification · Online Learning and Analytics · AI in Service Interactions
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Educational Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China
References (78)
Show source-provided references
1. Improving programming skills in engineering education through problem-based game projects with Scratch10.1016/j.compedu.2018.01.011 ↗
2. From critique to computational thinking: A peer-assessment-supported problem identification, flow definition, coding, and testing approach for computer programming instruction10.1177/07356331211060470 ↗
3. How does artificial intelligence teaching cultivate high school students’ computational thinking?—An empirical study based on AI case-driven Python programming teaching
4. An Exploration of a Hybrid Site Rotation Model to Support Differentiated Programming Instruction in Elementary Schools
5. Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis10.1111/bjet.13334 ↗
6. AI-generated feedback on writing: Insights into efficacy and ENL student preference10.1186/s41239-023-00425-2 ↗
7. Promoting self-regulation progress and knowledge construction in blended learning via ChatGPT-based learning aid10.1177/07356331231191125 ↗
8. Stathakarou, N., Nifakos, S., Karlgren, K., Konstantinidis, S.T., Bamidis, P.D., Pattichis, C.S., and Davoody, N. (2020). Students’ perceptions on chatbots’ potential and design characteristics in healthcare education. The Importance of Health Informatics in Public Health During a Pandemic, IOS Press.
9. Development of an educational chatbot system for enhancing students’ biology learning performance10.53106/160792642023032402006 ↗
10. Intelligent midwifery pedagogy: An example of the teaching practice of “Intelligent Socratic Conversation Robot”
11. Effects of a ChatGPT-based flipped learning guiding approach on learners’ courseware project performances and perceptions10.14742/ajet.8923 ↗
12. Hu, Y.H. (2024). Improving ethical dilemma learning: Featuring thinking aloud pair problem solving (TAPPS) and AI-assisted virtual learning companion. Educ. Inf. Technol., 1–21.10.1007/s10639-024-12754-4 ↗
13. Interacting with educational chatbots: A systematic review10.1007/s10639-022-11177-3 ↗
14. How competition in a game-based science learning environment influences students’ learning achievement, flow experience, and learning behavioral patterns
15. Digital natives, digital immigrants part 1
16. The games generations: How learners have changed
17. Zimmerman, B.J., and Schunk, D.H. (2011). Self-regulated learning and performance: An introduction and an overview. Handbook of Self-Regulation of Learning and Performance, Routledge.
18. The benefits of a challenge: Student motivation and flow experience in tablet-PC-game-based learning10.1080/10494820.2014.997248 ↗
19. Czikszentmihalyi, M. (1990). Flow: The Psychology of Optimal Experience, Harper & Row.
20. Yeoh, C.P., Li, C.T., and Hou, H.T. (2024). Game-based collaborative scientific inquiry learning using realistic context and inquiry process-based multidimensional scaffolding. Int. J. Sci. Educ., 1–23.10.1080/09500693.2024.2354944 ↗
21. Yang, Y.F., Lee, I.C., Tseng, C.C., and Lai, S.C. (2024). Developing students’ self-regulated learning strategies to facilitate vocabulary development in a digital game-based learning environment. J. Res. Technol. Educ., 1–20.10.1080/15391523.2024.2338092 ↗
22. Liang, H.Y., Hwang, G.J., Hsu, T.Y., and Yeh, J.Y. (2024). Effect of an AI-based chatbot on students’ learning performance in alternate reality game-based museum learning. Br. J. Educ. Technol.10.1111/bjet.13448 ↗
23. Information technology education: Historical evolution and design logic
24. Learning Styles in Compulsory Information Technology Curriculum: Digital Learning
25. Why and how to implement IT curriculum: New thinking in the perspective of digital transformation of education
26. Artificial intelligence chatbot behavior change model for designing artificial intelligence chatbots to promote physical activity and a healthy diet10.2196/22845 ↗
27. Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: The effects on learning performance and intrinsic motivation10.1177/07356331221077901 ↗
28. Yuan, Y. (2023). An empirical study of the efficacy of AI chatbots for English as a foreign language learning in primary education. Interact. Learn. Environ., 1–16.10.1080/10494820.2023.2282112 ↗
29. Development of mobile interactive courses based on an artificial intelligence chatbot on the communication software LINE10.1080/10494820.2021.1937230 ↗
30. Empowering ChatGPT with guidance mechanism in blended learning: Effect of self-regulated learning, higher-order thinking skills, and knowledge construction10.1186/s41239-024-00447-4 ↗
31. Effects of artificial Intelligence–Enabled personalized recommendations on learners’ learning engagement, motivation, and outcomes in a flipped classroom10.1016/j.compedu.2022.104684 ↗
32. Chatbots for language learning—Are they really useful? A systematic review of chatbot- supported language learning10.1111/jcal.12610 ↗
33. Alemdag, E. (2023). The effect of chatbots on learning: A meta-analysis of empirical research. J. Res. Technol. Educ., 1–23.10.1080/15391523.2023.2255698 ↗
34. Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness10.1016/j.compedu.2013.02.019 ↗
35. Towards the successful game-based learning: Detection and feedback to misconceptions is the key10.1016/j.compedu.2020.104033 ↗
36. Hwang, G.J., Chuang, W.H., and Hsia, L.H. (2024). Comprehending complex chemistry problems in a structured and enjoyable manner: A concept mapping-based contextual gaming approach. Educ. Inf. Technol., 1–23.10.1007/s10639-024-12615-0 ↗
37. The learning analytics of model-based learning facilitated by a problem-solving simulation game10.1007/s11251-018-9461-5 ↗
38. Promoting art appreciation performances and behaviors in effective and joyful contexts: A two-tier test-based digital gaming approach10.1016/j.compedu.2022.104706 ↗
39. Ng, D.T.K., Chen, X., Leung, J.K.L., and Chu, S.K.W. (2024). Fostering students’ AI literacy development through educational games: AI knowledge, affective and cognitive engagement. J. Comput. Assist. Learn.10.1111/jcal.13009 ↗
40. Li, X., and Li, R. (2024). How game-based learning supports the creativity of university students? The context of China. Innov. Educ. Teach. Int., 1–15.10.1080/14703297.2024.2351528 ↗
41. Amzalag, M., Kadusi, D., and Peretz, S. (2024). Enhancing Academic Achievement and Engagement Through Digital Game-Based Learning: An Empirical Study on Middle School Students. J. Educ. Comput. Res., 07356331241236937.10.1177/07356331241236937 ↗
42. Chen, C.H., and Chang, C.L. (2024). Effectiveness of AI-assisted game-based learning on science learning outcomes, intrinsic motivation, cognitive load, and learning behavior. Educ. Inf. Technol., 1–22.10.1007/s10639-024-12553-x ↗
43. Defining higher order thinking10.1080/00405849309543588 ↗
44. Applying a business simulation game in a flipped classroom to enhance engagement, learning achievement, and higher-order thinking skills10.1016/j.compedu.2022.104494 ↗
45. A study on the relationship between student’learning engagements and higher-order thinking skills in programming learning10.1016/j.tsc.2023.101369 ↗
46. A long-term experiment to investigate the relationships between high school students’ perceptions of mobile learning and peer interaction and higher-order thinking tendencies10.1007/s11423-017-9540-3 ↗
47. Exploring the structure of students’ scientific higher order thinking in science education10.1016/j.tsc.2022.100999 ↗
48. Computational thinking10.1145/1118178.1118215 ↗
49. Examining the key influencing factors on college students’ higher-order thinking skills in the smart classroom environment10.1186/s41239-020-00238-7 ↗
50. A short instrument for measuring students’ confidence with ‘key skills’ (SICKS): Development, validation and initial results10.1016/j.tsc.2020.100700 ↗
51. Song, C., and Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Front. Psychol., 14.10.3389/fpsyg.2023.1260843 ↗
52. Designing digital game-based learning for professional upskilling: A systematic literature review10.1016/j.compedu.2022.104518 ↗
53. The impact of educational chatbot on student learning experience10.1007/s10639-023-12166-w ↗
54. Space adventure game-based learning: How games and scaffolds affect eighth graders’ physics learning and game immersion10.1109/TLT.2023.3288879 ↗
55. The impacts of real competition and virtual competition in digital game-based learning10.1016/j.chb.2019.106171 ↗
56. Developing a digital game-based situated learning system for ocean ecology10.1007/978-3-540-69744-2_5 ↗
57. Effects of mobile learning time on students’ onception of collaboration, communication, complex problem–solving, meta–cognitive awareness and creativity10.1504/IJMLO.2014.067029 ↗
58. Trends and strategies for conducting effective STEM research and applications: A mobile and ubiquitous learning perspective10.1504/IJMLO.2020.106166 ↗
59. Pintrich, P.R., Smith, D.A.F., Garcia, T., and McKeachie, W.J. (1991). A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ), National Center for Research to Improve Postsecondary Teaching and Learning.
60. The ebb and flow of online learning10.1016/S0747-5632(04)00036-6 ↗
61. Effects of a concept mapping-based two-tier test strategy on students’ digital game-based learning performances and behavioral patterns10.1016/j.compedu.2021.104293 ↗
62. Effectiveness of an extended-reality interactive learning system in a dance training course10.1007/s10639-023-11883-6 ↗
63. Chen, Y.C., Hwang, G.J., and Lai, C.L. (2024). Motivating students to become self-regulatory learners: A gamified mobile self-regulated learning approach. Educ. Inf. Technol., 1–24.10.1007/s10639-024-12462-z ↗
64. Zimmerman, B.J., Bonner, S., and Kovach, R. (1996). Developing Self-Regulated Learners: Beyond Achievement to Self-Efficacy, American Psychological Association.10.1037/10213-000 ↗
65. Wouters, P., and van Oostendorp, H. (2016). Overview of instructional techniques to facilitate learning and motivation of serious games. Instructional Techniques to Facilitate Learning and Motivation of Serious Games, Springer.10.1007/978-3-319-39298-1 ↗
66. Implementing a theory-driven gamification model in higher education flipped courses: Effects on out-of-class activity completion and quality of artifacts10.1016/j.compedu.2018.06.018 ↗
67. Chien, C.C., Chan, H.Y., and Hou, H.T. (2024). Learning by playing with generative AI: Design and evaluation of a role-playing educational game with generative AI as scaffolding for instant feedback interaction. J. Res. Technol. Educ., 1–20.10.1080/15391523.2024.2338085 ↗
68. From gaming to computational thinking: An adaptive educational computer game-based learning approach10.1177/0735633120965919 ↗
69. Hsu, C.Y., and Wu, T.T. (2023). Application of Business Simulation Games in Flipped Classrooms to Facilitate Student Engagement and Higher-Order Thinking Skills for Sustainable Learning Practices. Sustainability, 15.10.20944/preprints202311.0082.v1 ↗
70. Enhancing student’s computational thinking skills with student-generated questions strategy in a game-based learning platform10.1016/j.compedu.2023.104794 ↗
71. Chang, C.C., Warden, C.A., Liang, C., and Lin, G.Y. (2018). Effects of digital game-based learning on achievement, flow and overall cognitive load. Australas. J. Educ. Technol., 34.10.14742/ajet.2961 ↗
72. Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning10.1016/j.chb.2015.07.045 ↗
73. Sun, L., Kangas, M., and Ruokamo, H. (2023). Game-based features in intelligent game-based learning environments: A systematic literature review. Interact. Learn. Environ., 1–17.10.1080/10494820.2023.2179638 ↗
74. Digital game elements, user experience and learning: A conceptual framework10.1007/s10639-018-9730-6 ↗
75. Self-regulation and academic achievement in elementary school children10.1002/cd.302 ↗
76. What makes high achievers different from low achievers? Self-regulated learners in the context of a high-stakes academic long-term task10.1016/j.lindif.2021.102085 ↗
77. High achiever students’ self regulated learning strategies, motivation towards mathematics, and their thinking styles
78. Effects of self-regulated learning strategy use on motivation in EFL writing: A comparison between high and low achievers in Hong Kong primary schools10.1515/applirev-2018-0085 ↗