Large Language Model-Powered Automated Assessment: A Systematic Review
Source abstract
Abstract
This systematic review investigates 49 peer-reviewed studies on Large Language Model-Powered Automated Assessment (LLMPAA) published between 2018 and 2024. Following PRISMA guidelines, studies were selected from Web of Science, Scopus, IEEE, ACM Digital Library, and PubMed databases. The analysis shows that LLMPAA has been widely applied in reading comprehension, language education, and computer science, primarily using essay and short-answer formats. While models such as GPT-4 and fine-tuned BERT often exhibit high agreement with human raters (e.g., QWK = 0.99, r = 0.95), other studies report lower agreement (e.g., ICC = 0.45, r = 0.38). LLMPAA offers benefits like efficiency, scalability, and personalized feedback. However, significant challenges remain, including bias, inconsistency, hallucination, limited explainability, dataset quality, and privacy concerns. These findings indicate that while LLMPAA technologies hold promise, their effectiveness varies by context. Human oversight is essential to ensure fair and reliable assessment outcomes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Machine Learning in Healthcare · Radiomics and Machine Learning in Medical Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Distance Education Application and Research, Ege University, İzmir 35040, Turkey
References (90)
Show source-provided references
1. Improving Automatic Short Answer Scoring Task Through a Hybrid Deep Learning Framework
2. Who’s the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers10.1177/07356331231191174 ↗
3. Automatic Short Answer Grading and Feedback Using Text Mining Methods10.1016/j.procs.2020.02.171 ↗
4. Yan, Y., and Liu, H. (2024). Ethical Framework for AI Education Based on Large Language Models. Educ. Inf. Technol., 1–19.10.1007/s10639-024-13241-6 ↗
5. An Overview of Automated Scoring of Essays
6. Using Active Learning Methods to Strategically Select Essays for Automated Scoring10.1111/emip.12537 ↗
7. The Eras and Trends of Automatic Short Answer Grading10.1007/s40593-014-0026-8 ↗
8. Henkel, O., Hills, L., Roberts, B., and Mcgrane, J. (2024). Can LLMs Grade Open Response Reading Comprehension Questions? An Empirical Study Using the ROARs Dataset. Int. J. Artif. Intell. Educ., 1–26.10.1007/s40593-024-00431-z ↗
9. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement
10. Mendonça, P.C., Quintal, F., and Mendonça, F. (2025). Evaluating LLMs for Automated Scoring in Formative Assessments. Appl. Sci., 15.10.3390/app15052787 ↗
11. Thurzo, A. (2025). Provable AI Ethics and Explainability in Medical and Educational AI Agents: Trustworthy Ethical Firewall. Electronics, 14.10.20944/preprints202502.2232.v1 ↗
12. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL HLT 2019—2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies—Proceedings of the Conference, Minneapolis, MN, USA.
13. Arici, N., Gerevini, A.E., Olivato, M., Putelli, L., Sigalini, L., and Serina, I. (2023). Real-World Implementation and Integration of an Automatic Scoring System for Workplace Safety Courses in Italian. Future Internet, 15.10.3390/fi15080268 ↗
14. Optimization of AES Using BERT and BiLSTM for Grading the Online Exams
15. An Empirical Analysis of BERT Embedding for Automated Essay Scoring
16. Bonthu, S., Sree, S.R., and Prasad, M.H.M.K. (2024). SPRAG: Building and Benchmarking a Short Programming-Related Answer Grading Dataset. Int. J. Data Sci. Anal., 1–13.10.1007/s41060-024-00576-z ↗
17. ChatGPT as an Automated Essay Scoring Tool in the Writing Classrooms: How It Compares with Human Scoring10.1007/s10639-024-12891-w ↗
18. An Optimized BERT Fine-Tuned Model Using an Artificial Bee Colony Algorithm for Automatic Essay Score Prediction10.7717/peerj-cs.2191 ↗
19. Dual-Scale BERT Using Multi-Trait Representations for Holistic and Trait-Specific Essay Grading10.4218/etrij.2023-0324 ↗
20. Automated Scoring of Short-Answer Questions: A Progress Report10.1080/08957347.2024.2386945 ↗
21. ChatGPT as an Instructor’s Assistant for Generating and Scoring Exams
22. Using Automated Procedures to Score Educational Essays Written in Three Languages10.1111/jedm.12406 ↗
23. Grévisse, C. (2024). LLM-Based Automatic Short Answer Grading in Undergraduate Medical Education. BMC Med. Educ., 24.10.1186/s12909-024-06026-5 ↗
24. Huang, F., Sun, X., Mei, A., Wang, Y., Ding, H., and Zhu, T. (2024). LLM Plus Machine Learning Outperform Expert Rating to Predict Life Satisfaction from Self-Statement Text. IEEE Trans. Comput. Soc. Syst.10.1109/TCSS.2024.3475413 ↗
25. Can AI-Assisted Essay Assessment Support Teachers? A Cross-Sectional Mixed-Methods Research Conducted at the University of Montenegro
26. A Comparative Analysis of the Rating of College Students’ Essays by ChatGPT versus Human Raters10.26803/ijlter.23.2.23 ↗
27. A Hybrid Approach for Automated Short Answer Grading10.1109/ACCESS.2024.3420890 ↗
28. Toward AI Grading of Student Problem Solutions in Introductory Physics: A Feasibility Study10.1103/PhysRevPhysEducRes.19.020163 ↗
29. Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading10.1007/s44163-024-00147-y ↗
30. Automated Essay Scoring Using Convolutional Neural Network Long Short-Term Memory with Mean of Question-Answer Encoding
31. A BERT-Based Automatic Scoring Model of Korean Language Learners’ Essay
32. Li, F., Xi, X., Cui, Z., Li, D., and Zeng, W. (2023). Automatic Essay Scoring Method Based on Multi-Scale Features. Appl. Sci., 13.10.3390/app13116775 ↗
33. Applying Large Language Models for Automated Essay Scoring for Non-Native Japanese
34. ChatGPT Analysis of Strengths and Weaknesses in English Writing and Their Implications
35. Enhanced BERT Approach to Score Arabic Essay’s Relevance to the Prompt
36. A Deep-Learning-Based Grading System (ASAG) for Reading Comprehension Assessment by Using Aphorisms as Open-Answer-Questions10.1007/s10639-023-11890-7 ↗
37. Automatic Essay Scoring for Arabic Short Answer Questions Using Text Mining Techniques
38. Exploring the Potential of Using an AI Language Model for Automated Essay Scoring10.1016/j.rmal.2023.100050 ↗
39. Beyond Semantic Distance: Automated Scoring of Divergent Thinking Greatly Improves with Large Language Models10.1016/j.tsc.2023.101356 ↗
40. Large Language Models and Automated Essay Scoring of English Language Learner Writing: Insights into Validity and Reliability10.1016/j.caeai.2024.100234 ↗
41. Quah, B., Zheng, L., Sng, T.J.H., Yong, C.W., and Islam, I. (2024). Reliability of ChatGPT in Automated Essay Scoring for Dental Undergraduate Examinations. BMC Med. Educ., 24.10.1186/s12909-024-05881-6 ↗
42. A Multitask Learning System for Trait-Based Automated Short Answer Scoring
43. Coherence-Based Automatic Short Answer Scoring Using Sentence Embedding10.1111/ejed.12684 ↗
44. Indonesian Automatic Short Answer Grading System10.11591/eei.v11i3.3531 ↗
45. The Revival of Essay-Type Questions in Medical Education: Harnessing Artificial Intelligence and Machine Learning10.29271/jcpsp.2024.05.595 ↗
46. Modeling Essay Grading with Pre-Trained BERT Features10.1007/s10489-024-05410-4 ↗
47. Automated Essay Scoring and Revising Based on Open-Source Large Language Models10.1109/TLT.2024.3396873 ↗
48. Harnessing LLMs for Multi-Dimensional Writing Assessment: Reliability and Alignment with Human Judgments10.1016/j.heliyon.2024.e34262 ↗
49. Can AI Provide Useful Holistic Essay Scoring?10.1016/j.caeai.2024.100255 ↗
50. Wang, Q., and Gayed, J.M. (2024). Effectiveness of Large Language Models in Automated Evaluation of Argumentative Essays: Finetuning vs. Zero-Shot Prompting. Comput. Assist. Lang. Learn.10.1080/09588221.2024.2371395 ↗
51. Wijanto, M.C., and Yong, H.S. (2024). Combining Balancing Dataset and SentenceTransformers to Improve Short Answer Grading Performance. Appl. Sci., 14.10.3390/app14114532 ↗
52. Automatic Short Answer Grading System in Indonesian Language Using BERT Machine Learning
53. Wu, Y., Henriksson, A., Nouri, J., Duneld, M., and Li, X. (2022). Beyond Benchmarks: Spotting Key Topical Sentences While Improving Automated Essay Scoring Performance with Topic-Aware BERT. Electronics, 12.10.3390/electronics12010150 ↗
54. Automatic Short-Answer Grading via BERT-Based Deep Neural Networks10.1109/TLT.2022.3175537 ↗
55. A Hierarchical BERT-Based Transfer Learning Approach for Multi-Dimensional Essay Scoring10.1109/ACCESS.2021.3110683 ↗
56. An Application of Many-Facet Rasch Measurement to Evaluate Automated Essay Scoring: A Case of ChatGPT-4.010.1016/j.rmal.2024.100133 ↗
57. Utilizing Large Language Models for EFL Essay Grading: An Examination of Reliability and Validity in Rubric-Based Assessments10.1111/bjet.13494 ↗
58. GPT Detectors Are Biased against Non-Native English Writers10.1016/j.patter.2023.100779 ↗
59. Lin, H., and Chen, Q. (2024). Artificial Intelligence (AI) -Integrated Educational Applications and College Students’ Creativity and Academic Emotions: Students and Teachers’ Perceptions and Attitudes. BMC Psychol., 12.10.1186/s40359-024-01979-0 ↗
60. Hackl, V., Müller, A.E., Granitzer, M., and Sailer, M. (2023). Is GPT-4 a Reliable Rater? Evaluating Consistency in GPT-4’s Text Ratings. Front. Educ., 8.10.3389/feduc.2023.1272229 ↗
61. Evaluating Large Language Models in Theory of Mind Tasks10.1073/pnas.2405460121 ↗
62. Using Cognitive Psychology to Understand GPT-310.1073/pnas.2218523120 ↗
63. Medical Teacher’s First ChatGPT’s Referencing Hallucinations: Lessons for Editors, Reviewers, and Teachers10.1080/0142159X.2023.2208731 ↗
64. ChatGPT Applications in Medical, Dental, Pharmacy, and Public Health Education: A Descriptive Study Highlighting the Advantages and Limitations10.52225/narra.v3i1.103 ↗
65. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions10.2196/48291 ↗
66. The Benefits and Challenges of ChatGPT: An Overview10.54097/fcis.v2i2.4465 ↗
67. ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education10.1016/j.lindif.2023.102274 ↗
68. A Survey on Large Language Model (LLM) Security and Privacy: The Good, The Bad, and The Ugly10.1016/j.hcc.2024.100211 ↗
69. Automatic Assessment of Text-Based Responses in Post-Secondary Education: A Systematic Review10.1016/j.caeai.2024.100206 ↗
70. Language Models Are Few-Shot Learners
71. Performance of ChatGPT on the Test of Understanding Graphs in Kinematics10.1103/PhysRevPhysEducRes.20.010109 ↗
72. Kaufmann, T., Weng, P., Kunshan, D., Bengs, V., and Hüllermeier, E. (2023). A Survey of Reinforcement Learning from Human Feedback. arXiv.
73. Atkinson, J., and Palma, D. (2025). An LLM-Based Hybrid Approach for Enhanced Automated Essay Scoring. Sci. Rep., 15.10.1038/s41598-025-87862-3 ↗
74. (2025, May 08). Introducing Claude 3.5 Sonnet\Anthropic. Available online: https://www.anthropic.com/news/claude-3-5-sonnet.
75. Prompt Engineering with ChatGPT: A Guide for Academic Writers10.1007/s10439-023-03272-4 ↗
76. A Comprehensive Survey of ChatGPT: Advancements, Applications, Prospects, and Challenges10.1016/j.metrad.2023.100022 ↗
77. Data Science and Its Relationship to Big Data and Data-Driven Decision Making10.1089/big.2013.1508 ↗
78. Powers, D.M.W. (2020). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. arXiv.
79. Weighted Kappa: Nominal Scale Agreement Provision for Scaled Disagreement or Partial Credit10.1037/h0026256 ↗
80. Mohammad, A.F., Clark, B., Agarwal, R., and Summers, S. (2023, January 24–27). LLM/GPT Generative AI and Artificial General Intelligence (AGI): The Next Frontier. Proceedings of the 2023 Congress in Computer Science, Computer Engineering, and Applied Computing, CSCE 2023, Las Vegas, NV, USA,.10.1109/CSCE60160.2023.00073 ↗
81. (2025, May 07). AI Principles|OECD. Available online: https://www.oecd.org/en/topics/ai-principles.html.
82. Shahriari, K., and Shahriari, M. (2017, January 21–22). IEEE Standard Review—Ethically Aligned Design: A Vision for Prioritizing Human Wellbeing with Artificial Intelligence and Autonomous Systems. Proceedings of the IHTC 2017—IEEE Canada International Humanitarian Technology Conference 2017, Toronto, ON, Canada.10.1109/IHTC.2017.8058187 ↗
83. Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep Learning, MIT Press.
84. Howard, J., and Ruder, S. (2018, January 15–20). Universal Language Model Fine-Tuning for Text Classification. Proceedings of the ACL 2018—56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Melbourne, Australia.10.18653/v1/P18-1031 ↗
85. Deep Reinforcement Learning from Human Preferences
86. Gal, Y., and Ghahramani, Z. (2016, January 19–24). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York, NY, USA.
87. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
88. Goslen, A., Kim, Y.J., Rowe, J., and Lester, J. (2024). LLM-Based Student Plan Generation for Adaptive Scaffolding in Game-Based Learning Environments. Int. J. Artif. Intell. Educ., 1–26.10.1007/s40593-024-00421-1 ↗
89. (2025, May 08). Grok 3 Beta—The Age of Reasoning Agents|XAI. Available online: https://x.ai/news/grok-3.
90. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., and Ruan, C. (2024). Deepseek-v3 technical report. arXiv.