Evaluating LLMs for Automated Scoring in Formative Assessments
Source abstract
Abstract
The increasing complexity and scale of modern education have revealed the shortcomings of traditional grading methods in providing consistent and scalable assessments. Advancements in artificial intelligence have positioned Large Language Models (LLMs) as robust solutions for automating grading tasks. This study systematically compared the grading performance of an open-source LLM (LLaMA 3.2) and a premium LLM (OpenAI GPT-4o) against human evaluators across diverse question types in the context of a computer programming subject. Using detailed rubrics, the study assessed the alignment between LLM-generated and human-assigned grades. Results revealed that while both LLMs align closely with human grading, equivalence testing demonstrated that the premium LLM achieves statistically and practically similar grading patterns, particularly for code-based questions, suggesting its potential as a reliable tool for educational assessments. These findings underscore the ability of LLMs to enhance grading consistency, reduce educator workload, and address scalability challenges in programming-focused assessments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Intelligent Tutoring Systems and Adaptive Learning · Educational Assessment and Pedagogy
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Exact Sciences and Engineering, Penteada University Campus, University of Madeira, 9000-082 Funchal, Portugal
Interactive Technologies Institute (ITI/LARSyS) and ARDITI, 9020-105 Funchal, Portugal
Funding
Fundação para a Ciência e a Tecnologia (FCT)
References (31)
Show source-provided references
1. Beyond Traditional Assessment: Exploring the Impact of Large Language Models on Grading Practices10.51219/JAIMLD/oluwole-fagbohun/19 ↗
2. A Meta Systematic Review of Artificial Intelligence in Higher Education: A Call for Increased Ethics, Collaboration, and Rigour10.1186/s41239-023-00436-z ↗
3. (2024). Automatic Assessment of Text-Based Responses in Post-Secondary Education: A Systematic Review. Comput. Educ. Artif. Intell., 6, 100206.10.1016/j.caeai.2024.100206 ↗
4. (2023). The Emergent Role of Artificial Intelligence, Natural Learning Processing, and Large Language Models in Higher Education and Research. Res. Social. Adm. Pharm., 19, 1236–1242.10.1016/j.sapharm.2023.05.016 ↗
5. Henkel, O., Hills, L., Roberts, B., and McGrane, J. (2024). Can LLMs Grade Open Response Reading Comprehension Questions? An Empirical Study Using the ROARs Dataset. Int. J. Artif. Intell. Educ.10.1007/s40593-024-00431-z ↗
6. Liu, M., and M’Hiri, F. (2024, January 20–23). Beyond Traditional Teaching: Large Language Models as Simulated Teaching Assistants in Computer Science. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, Portland, OR, USA.10.1145/3626252.3630789 ↗
7. Transforming Educational Assessment: Insights Into the Use of ChatGPT and Large Language Models in Grading
8. Kosar, T., Ostojić, D., Liu, Y.D., and Mernik, M. (2024). Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers. Mathematics, 12.10.3390/math12050629 ↗
9. Examining Science Education in ChatGPT: An Exploratory Study of Generative Artificial Intelligence10.1007/s10956-023-10039-y ↗
10. Zhai, X. (2025, March 02). ChatGPT User Experience: Implications for Education 2022. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4312418.10.2139/ssrn.4312418 ↗
11. Dorfner, F.J., Jürgensen, L., Donle, L., Mohamad, F.A., Bodenmann, T.R., Cleveland, M.C., Busch, F., Adams, L.C., Sato, J., and Schultz, T. (2024). Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports. arXiv.
12. Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O.A., and Cheungpasitporn, W. (2024). Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical Applications. Medicina, 60.10.3390/medicina60030445 ↗
13. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv.
14. Fung, S.C.E., Wong, M.F., and Tan, C.W. (2024, January 18–20). Automatic Feedback Generation on K-12 Students’ Data Science Education by Prompting Cloud-Based Large Language Models. Proceedings of the Eleventh ACM Conference on Learning @ Scale, Atlanta, GA, USA.10.1145/3657604.3664673 ↗
15. Artificial Intelligence Text Processing Using Retrieval-Augmented Generation: Applications in Business and Education Fields10.2478/picbe-2024-0018 ↗
16. Kochmar, E., Bexte, M., Burstein, J., Horbach, A., Laarmann-Quante, R., Tack, A., Yaneva, V., and Yuan, Z. (2024). Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics.
17. (2025, January 19). (PDF) Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring. Available online: https://www.researchgate.net/publication/382944163_Generative_Language_Models_with_Retrieval_Augmented_Generation_for_Automated_Short_Answer_Scoring.
18. A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science
19. Comparative Analysis of Open Source and Proprietary Large Language Models: Performance and Accessibility
20. Wang, Y., Wang, M., Manzoor, M.A., Liu, F., Georgiev, G.N., Das, R.J., and Nakov, P. (2024, January 12–16). Factuality of Large Language Models: A Survey. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.1088 ↗
21. Jury, B., Lorusso, A., Leinonen, J., Denny, P., and Luxton-Reilly, A. (February, January 29). Evaluating LLM-Generated Worked Examples in an Introductory Programming Course. Proceedings of the 26th Australasian Computing Education Conference, Sydney, NSW, Australia.
22. Wilcoxon-Mann-Whitney or t-Test? On Assumptions for Hypothesis Tests and Multiple Interpretations of Decision Rules10.1214/09-SS051 ↗
23. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses10.1177/1948550617697177 ↗
24. Improving Inferences About Null Effects with Bayes Factors and Equivalence Tests10.1093/geronb/gby065 ↗
25. (2024). Fine-Tuning ChatGPT for Automatic Scoring. Comput. Educ. Artif. Intell., 6, 100210.10.1016/j.caeai.2024.100210 ↗
26. Henkel, O., Hills, L., Boxer, A., Roberts, B., and Levonian, Z. (2024, January 18–20). Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education. Proceedings of the Eleventh ACM Conference on Learning @ Scale, Atlanta, GA, USA.10.1145/3657604.3664693 ↗
27. Mansour, W.A., Albatarni, S., Eltanbouly, S., and Elsayed, T. (2024, January 20–25). Can Large Language Models Automatically Score Proficiency of Written Essays?. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Orino, Italia.
28. Automatic Short Answer Grading for Finnish with ChatGPT
29. Grévisse, C. (2024). LLM-Based Automatic Short Answer Grading in Undergraduate Medical Education. BMC Med. Educ., 24.10.1186/s12909-024-06026-5 ↗
30. Automated Essay Scoring and Revising Based on Open-Source Large Language Models10.1109/TLT.2024.3396873 ↗
31. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., and Wang, P. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.