Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models
Source abstract
Abstract
Hyperparameter tuning is a critical function necessary for the effective deployment of most machine learning (ML) algorithms. It is used to find the optimal hyperparameter settings of an ML algorithm in order to improve its overall output performance. To this effect, several optimization strategies have been studied for fine-tuning the hyperparameters of many ML algorithms, especially in the absence of model-specific information. However, because most ML training procedures need a significant amount of computational time and memory, it is frequently necessary to build an optimization technique that converges within a small number of fitness evaluations. As a result, a simple deterministic selection genetic algorithm (SDSGA) is proposed in this article. The SDSGA was realized by ensuring that both chromosomes and their accompanying fitness values in the original genetic algorithm are selected in an elitist-like way. We assessed the SDSGA over a variety of mathematical test functions. It was then used to optimize the hyperparameters of two well-known machine learning models, namely, the convolutional neural network (CNN) and the random forest (RF) algorithm, with application on the MNIST and UCI classification datasets. The SDSGA’s efficiency was compared to that of the Bayesian Optimization (BO) and three other popular metaheuristic optimization algorithms (MOAs), namely, the genetic algorithm (GA), particle swarm optimization (PSO) and biogeography-based optimization (BBO) algorithms. The results obtained reveal that the SDSGA performed better than the other MOAs in solving 11 of the 17 known benchmark functions considered in our study. While optimizing the hyperparameters of the two ML models, it performed marginally better in terms of accuracy than the other methods while taking less time to compute.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Data Classification · Metaheuristic Optimization Algorithms Research · Advanced Multi-Objective Optimization Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Ahmadu Bello University, Zaria 810107, Nigeria
Advanced Internet of Things, Next, Generation Enterprises and Institutions, Council for Scientific and Industrial Research, Pretoria 0001, South Africa
Communications and Autonomous Systems Group, Robert Gordon University, Aberdeen AB10 7GJ, UK
Department of Electrical and Electronics Engineering, University of Jos, Jos 930222, Nigeria
References (50)
Show source-provided references
1. Khurana, D., Koli, A., Khatter, K., and Singh, S. (2017). Natural Language Processing: State of The Art, Current Trends and Challenges. arXiv.
2. Natural language processing: State of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine10.1016/j.jbi.2013.06.004 ↗
3. Optimal policy learning for disease prevention using reinforcement learning
4. Mendoza, H., Klein, A., Feurer, M., Springenberg, J.T., and Hutter, F. (2016). Towards Automatically-Tuned Neural Networks. Automated Machine Learning, Springer.
5. Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 11–14). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487629 ↗
6. Loshchilov, I., Schoenauer, M., and Sebag, M. (2013, January 6–10). BI-Population CMA-ES Algorithms with Surrogate Models and Line Searches. Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation, Amsterdam, The Netherlands.10.1145/2464576.2482696 ↗
7. Yu, T., and Zhu, H. (2020). Hyper-Parameter Optimization: A Review of Algorithms and Applications. arXiv.
8. Levesque, J.C., Durand, A., Gagne, C., and Sabourin, R. (2017, January 14–19). Bayesian optimization for conditional hyperparameter spaces. Proceedings of the International Joint Conference on Neural Networks, Anchorage, AK, USA.10.1109/IJCNN.2017.7965867 ↗
9. Random search for hyper-parameter optimization
10. Bergstra, J., Yamins, D., and Cox, D. (2013, January 24–29). Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms. Proceedings of the 12th Python in Science Conference, Austin, TX, USA.10.25080/Majora-8b375195-003 ↗
11. Hyperparameter optimization for machine learning models based on Bayesian optimization
12. Loshchilov, I., and Hutter, F. (2016). CMA-ES for Hyperparameter Optimization of Deep Neural Networks. arXiv.
13. Algorithms for hyper-parameter optimization
14. Wang, J., Xu, J., and Wang, X. (2018). Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning. arXiv.
15. Jin, H., Song, Q., and Hu, X. (2019, January 4–8). Auto-keras: An efficient neural architecture search system. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, AK, USA.10.1145/3292500.3330648 ↗
16. Practical Bayesian optimization of machine learning algorithms
17. Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., and Talwalkar, A. (2018). A System for Massively Parallel Hyperparameter Tuning. arXiv.
18. Hyperband: A novel bandit-based approach to hyperparameter optimization
19. Jamieson, K., and Talwalkar, A. (2016, January 9–11). Non-stochastic best arm identification and hyperparameter optimization. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain.
20. Fast Bayesian hyperparameter optimization on large datasets10.1214/17-EJS1335SI ↗
21. Evolutionary tuning of multiple SVM parameters10.1016/j.neucom.2004.11.022 ↗
22. Li, Y., and Zhang, Y. (2020). Hyper-parameter estimation method with particle swarm optimization. arXiv.
23. Bacanin, N., Bezdan, T., Tuba, E., Strumberger, I., and Tuba, M. (2020). Optimizing convolutional neural network hyperparameters by enhanced swarm intelligence metaheuristics. Algorithms, 13.10.3390/a13030067 ↗
24. Heuristic hyperparameter optimization of deep learning models for genomic prediction
25. Lorenzo, P.R., Nalepa, J., Ramos, L.S., and Pastor, J.R. (2017, January 15–19). Hyper-parameter selection in deep neural networks using parallel particle swarm optimization. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Berlin, Germany.10.1145/3067695.3084211 ↗
26. Mantovani, R.G., Horvath, T., Cerri, R., Vanschoren, J., and De Carvalho, A.C. (2016, January 9–12). Hyper-Parameter Tuning of a Decision Tree Induction Algorithm. Proceedings of the 2016 5th Brazilian Conference on Intelligent Systems (BRACIS), Recife, Brazil.10.1109/BRACIS.2016.018 ↗
27. Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics10.1140/epjc/s10052-021-08950-y ↗
28. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., and Hutter, F. (2019). Auto-sklearn: Efficient and Robust Automated Machine Learning. Automated Machine Learning, Springer International Publishing.10.1007/978-3-030-05318-5_6 ↗
29. Kotthoff, L., Thornton, C., Hoos, H.H., Hutter, F., and Leyton-Brown, K. (2019). Auto-WEKA. Automated Machine Learning, Springer International Publishing.10.1007/978-3-030-05318-5_4 ↗
30. Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL10.1109/TPAMI.2021.3067763 ↗
31. Genetic Algorithm with Elitist Model and ITS Convergence10.1142/S0218001496000438 ↗
32. Genetic Algorithm with adaptive elitist-population strategies for multimodal function optimization10.1016/j.asoc.2010.06.017 ↗
33. A novel Clustering based Genetic Algorithm for route optimization
34. An optimized routing algorithm for vehicle ad-hoc networks
35. Allawi, Z.T., Ibraheem, I.K., and Humaidi, A.J. (2019). Fine-tuning meta-heuristic algorithm for global optimization. Processes, 7.10.3390/pr7100657 ↗
36. Comparison of Meta-heuristic Algorithms on Benchmark Functions10.33793/acperpro.02.03.41 ↗
37. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
38. Frank, A., and Asuncion, A. (2010). UCI Machine Learning Repository, University of California.
39. Genetic algorithms for modelling and optimisation10.1016/j.cam.2004.07.034 ↗
40. and Shi, Y. (2001, January 27–30). Eberhart; Shi, Y. Particle swarm optimization: Developments, applications and resources. Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546), Seoul, Korea.
41. Biogeography-Based Optimization: A 10-Year Review10.1109/TETCI.2017.2739124 ↗
42. Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN’95—International Conference on Neural Networks, Perth, WA, Australia.
43. Genetic algorithms: Concepts and applications [in engineering design]10.1109/41.538609 ↗
44. Elitist Genetic Algorithm Based Energy Balanced Routing Strategy to Prolong Lifetime of Wireless Sensor Networks
45. Biogeography-Based Optimization10.1109/TEVC.2008.919004 ↗
46. Classification and regression trees10.1002/widm.8 ↗
47. Oblique Decision Tree Ensemble via Multisurface Proximal Support Vector Machine10.1109/TCYB.2014.2366468 ↗
48. Menze, B.H., Kelm, B.M., Splitthoff, D.N., Koethe, U., and Hamprecht, F.A. (2011). On Oblique Random Forests, Springer.10.1007/978-3-642-23783-6_29 ↗
49. Heterogeneous oblique random forest10.1016/j.patcog.2019.107078 ↗
50. Multinomial random forest10.1016/j.patcog.2021.108331 ↗