Implementing AutoML in Educational Data Mining for Prediction Tasks
Source abstract
Abstract
Educational Data Mining (EDM) has emerged over the last two decades, concerning with the development and implementation of data mining methods in order to facilitate the analysis of vast amounts of data originating from a wide variety of educational contexts. Predicting students’ progression and learning outcomes, such as dropout, performance and course grades, is regarded among the most important tasks of the EDM field. Therefore, applying appropriate machine learning algorithms for building accurate predictive models is of outmost importance for both educators and data scientists. Considering the high-dimensional input space and the complexity of machine learning algorithms, the process of building accurate and robust learning models requires advanced data science skills, while is time-consuming and error-prone in most cases. In addition, choosing the proper method for a given problem formulation and configuring the optimal parameters’ values for a specific model is a demanding task, whilst it is often very difficult to understand and explain the produced results. In this context, the main purpose of the present study is to examine the potential use of advanced machine learning strategies on educational settings from the perspective of hyperparameter optimization. More specifically, we investigate the effectiveness of automated Machine Learning (autoML) for the task of predicting students’ learning outcomes based on their participation in online learning platforms. At the same time, we limit the search space to tree-based and rule-based models in order to achieving transparent and interpretable results. To this end, a plethora of experiments were carried out, revealing that autoML tools achieve consistently superior results. Hopefully our work will help nonexpert users (e.g., educators and instructors) in the field of EDM to conduct experiments with appropriate automated parameter configurations, thus achieving highly accurate and comprehensible results.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Online Learning and Analytics · Machine Learning and Data Classification · Data Stream Mining Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mathematics, University of Patras, 26504 Rio Patras, Greece
References (85)
Show source-provided references
1. Data mining in education10.1002/widm.1075 ↗
2. Educational data mining applications and tasks: A survey of the last 10 years10.1007/s10639-017-9616-z ↗
3. Educational data mining: A review of the state of the art10.1109/TSMCC.2010.2053532 ↗
4. Bousbia, N., and Belamri, I. (2014). Which Contribution Does EDM Provide to Computer-Based Learning Environments?. Educational Data Mining, Springer.10.1007/978-3-319-02738-8_1 ↗
5. Educational data science in massive open online courses10.1002/widm.1187 ↗
6. Wolff, A., Zdrahal, Z., Herrmannova, D., and Knoth, P. (2014). Predicting student performance from combined data sources. Educational Data Mining, Springer.10.1007/978-3-319-02738-8_7 ↗
7. Academic analytics: A new tool for a new era
8. Educational data mining: A survey from 1995 to 200510.1016/j.eswa.2006.04.005 ↗
9. Big Data and analytics in higher education: Opportunities and challenges10.1111/bjet.12230 ↗
10. Bergstra, J.S., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems, Curran Associates Inc.
11. Snoek, J., Larochelle, H., and Adams, R.P. (2012). Practical bayesian optimization of machine learning algorithms. NIPS’12 Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates Inc.
12. Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 11–14). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487629 ↗
13. Bergstra, J., Yamins, D., and Cox, D.D. (2013, January 21–25). Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. Proceedings of the 12th Python in Science Conference, Brussels, Belgium.10.25080/Majora-8b375195-003 ↗
14. Galitsky, B. (2018, January 26–28). Customers’ Retention Requires an Explainability Feature in Machine Learning Systems They Use. Proceedings of the 2018 AAAI Spring Symposium Series, Palo Alto, CA, USA.
15. Performance of classification models from a user perspective10.1016/j.dss.2011.01.013 ↗
16. Došilović, F.K., Brčić, M., and Hlupić, N. (2018). Explainable artificial intelligence: A survey. 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), IEEE.10.23919/MIPRO.2018.8400040 ↗
17. Hämäläinen, W., and Vinni, M. (2010). Classifiers for educational data mining. Handbook of Educational Data Mining, CRC Press.10.1201/b10274-7 ↗
18. Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS10.1109/TLT.2016.2616312 ↗
19. Predicting student failure at school using genetic programming and different data mining approaches with high dimensional and imbalanced data10.1007/s10489-012-0374-8 ↗
20. Moreno-Marcos, P.M., Alario-Hoyos, C., Muñoz-Merino, P.J., and Kloos, C.D. (2018). Prediction in MOOCs: A review and future research directions. IEEE Trans. Learn. Technol., 12.10.1109/TLT.2018.2856808 ↗
21. Modeling and predicting students’ academic performance using data mining techniques10.5815/ijmecs.2016.11.05 ↗
22. Mining educational data to predict student’s academic performance using ensemble methods10.14257/ijdta.2016.9.8.13 ↗
23. Classification and prediction based data mining algorithms to predict slow learners in education sector10.1016/j.procs.2015.07.372 ↗
24. Guo, B., Zhang, R., Xu, G., Shi, C., and Yang, L. (2015, January 27–29). Predicting students performance in educational data mining. Proceedings of the 2015 International Symposium on Educational Technology (ISET), Wuhan, China.10.1109/ISET.2015.33 ↗
25. Educational data mining & students’ performance prediction
26. Evaluating the effectiveness of educational data mining techniques for early prediction of students’ academic failure in introductory programming courses10.1016/j.chb.2017.01.047 ↗
27. Analyzing undergraduate students’ performance using educational data mining10.1016/j.compedu.2017.05.007 ↗
28. Kostopoulos, G., Kotsiantis, S., and Pintelas, P. (2015). Predicting student performance in distance higher education using semi-supervised techniques. Model and Data Engineering, Springer.10.1145/2801948.2802013 ↗
29. Predicting student performance using personalized analytics10.1109/MC.2016.119 ↗
30. A machine learning approach for tracking and predicting student performance in degree programs10.1109/JSTSP.2017.2692560 ↗
31. Strecht, P., Cruz, L., Soares, C., Mendes-Moreira, J., and Abreu, R. (2015, January 26–29). A Comparative Study of Classification and Regression Algorithms for Modelling Students’ Academic Performance. Proceedings of the 8th International Conference on Educational Data Mining, Madrid, Spain.
32. Meier, Y., Xu, J., Atan, O., and van der Schaar, M. (2015, January 14–17). Personalized grade prediction: A data mining approach. Proceedings of the 2015 IEEE International Conference on Data Mining, Atlantic City, NJ, USA.10.1109/ICDM.2015.54 ↗
33. Sweeney, M., Rangwala, H., Lester, J., and Johri, A. (2016). Next-term student performance prediction: A recommender systems approach. arXiv.10.1109/BigData.2015.7363847 ↗
34. A Semi-Supervised Regression Algorithm for Grade Prediction of Students in Distance Learning Courses10.1142/S0218213019400013 ↗
35. Tsiakmaki, M., Kostopoulos, G., Koutsonikos, G., Pierrakeas, C., Kotsiantis, S., and Ragos, O. (2018, January 23–25). Predicting University Students’ Grades Based on Previous Academic Achievements. Proceedings of the 2018 9th International Conference on Information, Intelligence, Systems and Applications (IISA), Zakynthos, Greece.10.1109/IISA.2018.8633618 ↗
36. Early dropout prediction using data mining: A case study with high school students10.1111/exsy.12135 ↗
37. Zhang, Y., Oussena, S., Clark, T., and Kim, H. (2010, January 8–12). Use Data Mining to Improve Student Retention in Higher Education-A Case Study. Proceedings of the 12th International Conference on Enterprise Information Systems, Volume 1, DISI, Funchal, Madeira, Portugal.
38. A comparative analysis of machine learning techniques for student retention management10.1016/j.dss.2010.06.003 ↗
39. Dropout prediction in e-learning courses through the combination of machine learning techniques10.1016/j.compedu.2009.05.010 ↗
40. Superby, J.-F., Vandamme, J.P., and Meskens, N. (2006, January 26–30). Determination of factors influencing the achievement of the first-year university students using data mining methods. Proceedings of the Workshop on Educational Data Mining, Jhongli, Taiwan.
41. Estimating student retention and degree-completion time: Decision trees and neural networks vis-à-vis regression
42. Kostopoulos, G., Kotsiantis, S., and Pintelas, P. (2015, January 1–3). Estimating student dropout in distance higher education using semi-supervised techniques. Proceedings of the 19th Panhellenic Conference on Informatics, Athens, Greece.10.1145/2801948.2802013 ↗
43. Rao, S.S. (2009). Engineering Optimization: Theory and Practice, John Wiley & Sons.10.1002/9780470549124 ↗
44. Brochu, E. (2010). Interactive Bayesian Optimization: Learning User Preferences for Graphics and Animation, University of British Columbia.
45. Feurer, M., and Hutter, F. (2019). Hyperparameter Optimization. Automated Machine Learning: Methods, Systems, Challenges, Springer.10.1007/978-3-030-05318-5_1 ↗
46. Brochu, E., Cora, V.M., and de Freitas, N. (2010). A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv.
47. Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning-Methods, Systems, Challenges, Springer.10.1007/978-3-030-05318-5 ↗
48. Random search for hyper-parameter optimization
49. Gradient-based optimization of hyperparameters10.1162/089976600300015187 ↗
50. The racing algorithm: Model selection for lazy learners10.1023/A:1006556606079 ↗
51. Simon, D. (2013). Evolutionary Optimization Algorithms, Wiley.
52. A novel LS-SVMs hyper-parameter selection based on particle swarm optimization10.1016/j.neucom.2008.04.027 ↗
53. Dewancker, I., McCourt, M., and Clark, S. (2019, June 12). Bayesian Optimization Primer; SigOpt. Available online: https://app.sigopt.com/static/pdf/SigOpt_Bayesian_Optimization_Primer.pdf.
54. Beyond manual tuning of hyperparameters10.1007/s13218-015-0381-0 ↗
55. Taking the human out of the loop: A review of bayesian optimization10.1109/JPROC.2015.2494218 ↗
56. Williams, C.K.I., and Rasmussen, C.E. (2006). Gaussian Processes for Machine Learning, MIT Press.
57. Bergstra, J., Yamins, D., and Cox, D.D. (2013, January 16–21). Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. Proceedings of the 30th International Conference on International Conference on Machine Learning, Atlanta, GA, USA.
58. Breiman, L., Friedman, J., Olshen, R., and Stone, C. (1984). Classification and Regression Trees, Wadsworth Int. Group.
59. Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2011, January 17–21). Sequential model-based optimization for general algorithm configuration. Proceedings of the International Conference on Learning and Intelligent Optimization, Rome, Italy.10.1007/978-3-642-25566-3_40 ↗
60. Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., and Leyton-Brown, K. (2013, January 10). Towards an empirical foundation for assessing bayesian optimization of hyperparameters. Proceedings of the NIPS Workshop on Bayesian Optimization in Theory and Practice, Lake Tahoe, NV, USA.
61. Efficient global optimization of expensive black-box functions10.1023/A:1008306431147 ↗
62. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise10.1115/1.3653121 ↗
63. Srinivas, N., Krause, A., Kakade, S.M., and Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv.
64. Clark, S., Liu, E., Frazier, P., Wang, J., Oktay, D., and Vesdapunt, N. (2019, June 12). MOE: A Global, Black Box Optimization Engine for Real World Metric Optimization. Available online: https://github.com/Yelp/MOE.
65. Extremely randomized trees10.1007/s10994-006-6226-1 ↗
66. Jeni, L.A., Cohn, J.F., and de la Torre, F. (2013, January 2–5). Facing imbalanced data–recommendations for the use of performance metrics. Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland.10.1109/ACII.2013.47 ↗
67. Ling, C.X., Huang, J., and Zhang, H. (2003, January 11–13). AUC: A better measure than accuracy in comparing learning algorithms. Proceedings of the Conference of the Canadian Society for Computational Studies of Intelligence, Halifax, NS, Canada.10.1007/3-540-44886-1_25 ↗
68. Provost, F., and Fawcett, T. (1997, January 14–17). Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions. Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, Newport Beach, CA, USA.
69. Frank, E., Hall, M.A., and Witten, I.H. (2016). The WEKA Workbench. Online Appendix for Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann. [4th ed.].
70. Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA
71. Witten, I.H., Frank, E., Hall, M.A., and Pal, C.J. (2016). Data Mining: Practical machine learning tools and techniques, Morgan Kaufmann.
72. John, G.H., and Langley, P. (1995, January 18–20). Estimating continuous distributions in Bayesian classifiers. Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada.
73. Eibe, F., and Witten, I.H. (1998). Generating Accurate Rule Sets without Global Optimization, University of Waikato, Department of Computer Science.
74. Holmes, G., Hall, M., and Prank, E. (1999, January 6–10). Generating rule sets from model trees. Proceedings of the Australasian Joint Conference on Artificial Intelligence, Sydney, Australia.10.1007/3-540-46695-9_1 ↗
75. Random forests10.1023/A:1010933404324 ↗
76. Schoelkopf, B., Burges, C., and Smola, A. (1998). Fast Training of Support Vector Machines using Sequential Minimal Optimization. Advances in Kernel Methods—Support Vector Learning, MIT Press.
77. Improvements to Platt’s SMO Algorithm for SVM Classifier Design10.1162/089976601300014493 ↗
78. Hastie, T., and Tibshirani, R. (1998). Classification by Pairwise Coupling. Advances in Neural Information Processing Systems, MIT Press.10.1214/aos/1028144844 ↗
79. Shevade, S.K., Keerthi, S.S., Bhattacharyya, C., and Murthy, K.R.K. (2000). Improvements to the SMO Algorithm for SVM Regression. IEEE Transactions on Neural Netw., 11.10.1109/72.870050 ↗
80. Smola, A.J., and Schoelkopf, B. (1998). A Tutorial on Support Vector Regression, Kluwer Academic Publishers.
81. Instance-based learning algorithms10.1007/BF00153759 ↗
82. Bagging predictors10.1007/BF00058655 ↗
83. Kim, B.-H., Vizitei, E., and Ganapathi, V. (2018). GritNet: Student performance prediction with deep learning. arXiv.
84. Caruana, R., and Niculescu-Mizil, A. (2006, January 25–29). An empirical comparison of supervised learning algorithms. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143865 ↗
85. No free lunch theorems for optimization10.1109/4235.585893 ↗