An Early Warning System to Detect At-Risk Students in Online Higher Education
Source abstract
Abstract
Artificial intelligence has impacted education in recent years. Datafication of education has allowed developing automated methods to detect patterns in extensive collections of educational data to estimate unknown information and behavior about the students. This research has focused on finding accurate predictive models to identify at-risk students. This challenge may reduce the students’ risk of failure or disengage by decreasing the time lag between identification and the real at-risk state. The contribution of this paper is threefold. First, an in-depth analysis of a predictive model to detect at-risk students is performed. This model has been tested using data available in an institutional data mart where curated data from six semesters are available, and a method to obtain the best classifier and training set is proposed. Second, a method to determine a threshold for evaluating the quality of the predictive model is established. Third, an early warning system has been developed and tested in a real educational setting being accurate and useful for its purpose to detect at-risk students in online higher education. The stakeholders (i.e., students and teachers) can analyze the information through different dashboards, and teachers can also send early feedback as an intervention mechanism to mitigate at-risk situations. The system has been evaluated on two undergraduate courses where results shown a high accuracy to correctly detect at-risk students.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Online Learning and Analytics · Anomaly Detection Techniques and Applications · Software System Performance and Reliability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
eLearn Center, Universitat Oberta de Catalunya, 08018 Barcelona, Spain
Faculty of Computer Science, Multimedia and Telecommunications, Universitat Oberta de Catalunya, 08018 Barcelona, Spain
Open Education Faculty, Yunus Emre Campus, Anadolu University, 26470 Eskisehir, Turkey
Funding
Universitat Oberta de Catalunya
References (67)
Show source-provided references
1. Craig, S.D. (2018). Tutoring and Intelligent Tutoring Systems, Nova Science Publishers, Incorporated.
2. Karadeniz, A., Baneres, D., Rodríguez, M.E., and Guerrero-Roldán, A.E. (2019, January 16–18). Enhancing ICT Personalized Education through a Learning Intelligent System. Proceedings of the Online, Open and Flexible Higher Education Conference, Madrid, Spain.
3. An Early Feedback Prediction System for Learners At-risk within a First-year Higher Education Subject10.1109/TLT.2019.2912167 ↗
4. Zawacki-Richter, O., Marín, V.I., Bond, M., and Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education—Where are the educators?. Int. J. Technol. High. Educ., 16.10.1186/s41239-019-0171-0 ↗
5. Rastrollo-Guerrero, J., Gámez-Pulido, J., and Durán-Domínguez, A. (2020). Analyzing and Predicting Students’ Performance by Means of Machine Learning: A Review. Appl. Sci., 10.10.3390/app10031042 ↗
6. Student engagement predictions in an e-learning system and their impact on student course assessment scores10.1155/2018/6347186 ↗
7. Early warning system as a predictor for student performance in higher education blended courses10.1080/03075079.2018.1466872 ↗
8. Lee, S., and Chung, J. (2019). The Machine Learning-Based Dropout Early Warning System for Improving the Performance of Dropout Prediction. Appl. Sci., 9.10.3390/app9153093 ↗
9. Exploring student characteristics of retention that lead to graduation in higher educationusing data mining models10.2190/CS.16.4.e ↗
10. Hammond, T., Prasad, M., and Stepanova, A. (2019). Teaching Engineering Students Freehand Sketching with an Intelligent Tutoring System. Inspiring Students with Digital Ink, Springer. [1st ed.].10.1007/978-3-030-17398-2 ↗
11. Motivation matters: Interactions between achievement goals and agent scaffolding for self-regulated learning within an intelligent tutoring system10.1016/j.chb.2015.05.041 ↗
12. Bydžovská, H. (July, January 29). Course Enrollment Recommender System. Proceedings of the 9th International Conference on Educational Data Mining, Raleigh, NC, USA.
13. Student performance prediction and optimal course selection: An MDP approach10.1007/978-3-319-74781-1_3 ↗
14. Analyzing student performance using sparse data of core bachelor courses
15. Mishra, T., Kumar, D., and Gupta, D.S. (2014, January 8–9). Mining students data for performance prediction. Proceedings of the Fourth International Conference on Advanced Computing & Communication Technologies, Rohtak, India.10.1109/ACCT.2014.105 ↗
16. Predicting academic performance by data mining methods10.1080/09645290701409939 ↗
17. Pistilli, M.D., and Arnold, K.E. (May, January 29). Course signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, Vancouver, BC, Canada.
18. Identifying significant indicators using LMS data to predict course achievement in online learning10.1016/j.iheduc.2015.11.003 ↗
19. Saura, J., Reyes-Menendez, A., and Bennett, D. (2019). How to Extract Meaningful Insights from UGC: A Knowledge-Based Method Applied to Education. Appl. Sci., 9.10.3390/app9214603 ↗
20. Predicting students’ final performance from participation in on-line discussion forums10.1016/j.compedu.2013.06.009 ↗
21. A multivariate approach to predicting student outcomes in web-enabled blended learning courses10.1016/j.iheduc.2015.05.002 ↗
22. Micro-analytics for student performance prediction leveraging fine-grained learning analytics to predict performance
23. Models for early prediction of at-risk students in a course using standards-based grading10.1016/j.compedu.2016.09.005 ↗
24. Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success10.1016/j.iheduc.2015.10.002 ↗
25. Mining LMS data to develop an early warning system for educators: A proof of concept10.1016/j.compedu.2009.09.008 ↗
26. Utilizing student activity patterns to predict performance10.1186/s41239-017-0044-3 ↗
27. Improving Early Warning Systems with Categorized Course Resource Usage10.18608/jla.2016.33.13 ↗
28. Learning at distance: Effects of interaction traces on academic achievement10.1016/j.compedu.2015.07.002 ↗
29. Predicting students’ marks from Moodle logs using neural network models
30. Contrasting prediction methods for early warning systems at undergraduate level10.1016/j.iheduc.2018.02.001 ↗
31. Predicting student performance from combined data sources10.1007/978-3-319-02738-8_7 ↗
32. Akçapınar, G., Altun, A., and Aşkar, P. (2019). Using learning analytics to develop early-warning system for at-risk students. Int. J. Educ. Technol. High. Educ., 16.10.1186/s41239-019-0172-z ↗
33. Buschetto Macarini, L.A., Cechinel, C., Batista Machado, M.F., Faria Culmant Ramos, V., and Munoz, R. (2019). Predicting Students Success in Blended Learning—Evaluating Different Interactions Inside Learning Management Systems. Appl. Sci., 9.10.3390/app9245523 ↗
34. Analytics of Communities of Inquiry: Effects of Learning Technology Use on Cognitive Presence in Asynchronous Online Discussions10.1016/j.iheduc.2015.06.002 ↗
35. Early dropout prediction using data mining: A case study with high school students10.1111/exsy.12135 ↗
36. Dropout prediction in e-learning courses through the combination of machine learning techniques10.1016/j.compedu.2009.05.010 ↗
37. Srilekshmi, M., Sindhumol, S., Shiffon, C., and Kamal, B. (December, January 28). Learning analytics to identify students at-risk in MOOCs. Proceedings of the IEEE 8th International Conference on Technology for Education, Mumbai, India.
38. Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization10.1016/j.chb.2015.12.007 ↗
39. Falkner, N.J., and Falkner, K.E. (2012, January 9–11). A fast measure for identifying at-risk students in computer science. Proceedings of the Ninth Annual International Conference on International Computing Education Research, Auckland, New Zealand.10.1145/2361276.2361288 ↗
40. Larusson, J., and White, B. (2014). A learning management system-based early warning system for academic advising in undergraduate engineering. Learning Analytics, Springer International Publisher. [1st ed.].10.1007/978-1-4614-3305-7 ↗
41. A Novel Predictive Modeling System to Analyze Students at Risk of Academic Failure
42. Developing early warning Systems to predict students’ online learning performance10.1016/j.chb.2014.04.002 ↗
43. Early alert of academically at-risk students: An open source analytics initiative10.18608/jla.2014.11.3 ↗
44. Vasquez, H., and Azarbayejani, M. (2015, January 21–24). Early identification of at-risk students in a lower-level engineering gatekeeper course. Proceedings of the 2015 IEEE Frontiers in Education Conference (FIE), El Paso, TX, USA.10.1109/FIE.2015.7344361 ↗
45. Supporting first generation college freshmen with small group intervention
46. Sunstein, C.R. (2020, May 24). Which Nudges Do People Like? A National Survey. Available online: https://ssrn.com/abstract=2619899.10.4337/9781785367854.00028 ↗
47. Nudging in education10.1016/j.econedurev.2018.03.008 ↗
48. Minguillón, J., Conesa, J., Rodríguez, M.E., and Santanach, F. (2018). Learning analytics in practice: Providing e-learning researches and practitioners with activity data. Frontiers of Cyberlearning: Emerging Technologies for Teaching and Learning, Springer.10.1007/978-981-13-0650-1_8 ↗
49. Drachsler, H., Hoel, T., Scheffel, M., Kismihók, G., Berg, A., Ferguson, R., and Manderveld, J. (2015, January 16–20). Ethical and privacy issues in the application of learning analytics. Proceedings of the 5th International Conference on Learning Analytics and Knowledge, ACM, Poughkeepsie, NY, USA.10.1145/2883851.2883933 ↗
50. (2020, May 24). UOC Virtual Campus. Available online: https://www.uoc.edu/estudiant/portal/guia/en/com_estudia/campus_virtual/index.html.
51. AUC and accuracy in evaluating learning algorithms10.1109/TKDE.2005.50 ↗
52. Scikit-learn: Machine learning in Python
53. Core Team (2020, May 24). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Available online: https://www.R-project.org/.
54. Chambers, J.M., and Hastie, T.J. (1992). Local regression models. Statistical Models in S, Chapman and Hall.
55. Baneres, D., Karadeniz, A., Guerrero-Roldán, A.E., and Rodríguez, M.E. (2020, January 5–6). A predictive system for supporting at-risk students’ identification. Proceedings of the Future Technologies Conference 2020, Vancouver, BC, Canada.10.1007/978-3-030-63128-4_67 ↗
56. Oates, B.J. (2005). Researching Information Systems and Computing, SAGE.
57. Vaishnavi, V., and Kuechler, W. (2020, May 24). Design Research in Information Systems. Available online: http://www.desrist.org/design-research-in-information-systems/.
58. (2020, May 24). UOC Performance and Satisfaction Indicators. Available online: https://www.uoc.edu/portal/en/qualitat/resultats/resultats-rendiment/index.html.
59. Identifying at-risk students for early interventions: A time-series clustering approach
60. Rodríguez, M.E., Guerrero-Roldán, A.E., Baneres, D., and Karadeniz, A. (2019, January 9–11). Towards an intervention mechanism for supporting learners performance in online learning. Proceedings of the 12th International Conference of Education, Research and Innovation, Sevilla, Spain.10.21125/iceri.2019.1244 ↗
61. Historical Notes on the Wilcoxon Unpaired Two-Sample Test10.1080/01621459.1957.10501395 ↗
62. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics10.1016/j.ins.2013.07.007 ↗
63. Sisovic, S., Matetic, M., and Bakaric, M.B. (2016, January 21–23). Clustering of imbalanced moodle data for early alert of student failure. Proceedings of the IEEE 14th International Symposium on Applied Machine Intelligence and Informatics, Herl’any, Slovakia.10.1109/SAMI.2016.7423001 ↗
64. Using early warning signs to predict academic risk in interactive, blended teaching environments
65. (2020, May 23). UOC Ethical Committee. Available online: https://research.uoc.edu/portal/en/ri/activitat-rdi/comite-etica/funcions/index.html.
66. School’s Out, But Class’ On’, The Largest Online Education in the World Today: Taking China’s Practical Exploration During The COVID-19 Epidemic Prevention and Control As an Example10.15354/bece.20.ar023 ↗
67. Clarizia, F., Colace, F., Lombardi, M., Pascale, F., and Santaniello, D. (2018, January 29–31). Chatbot: An education support system for student. Proceedings of the International Symposium on Cyberspace Safety and Security, Amalfi, Italy.10.1007/978-3-030-01689-0_23 ↗