An Enhanced Proximal Policy Optimization-Based Reinforcement Learning Method with Random Forest for Hyperparameter Optimization
Source abstract
Abstract
For most machine learning and deep learning models, the selection of hyperparameters has a significant impact on the performance of the model. Therefore, deep learning and data analysis experts have to spend a lot of time on hyperparameter tuning when building a model for accomplishing a task. Although there are many algorithms used to solve hyperparameter optimization (HPO), these methods require the results of the actual trials at each epoch to help perform the search. To reduce the number of trials, model-based reinforcement learning adopts multilayer perceptron (MLP) to capture the relationship between hyperparameter settings and model performance. However, MLP needs to be carefully designed because there is a risk of overfitting. Thus, we propose a random forest-enhanced proximal policy optimization (RFEPPO) reinforcement learning algorithm to solve the HPO problem. In addition, reinforcement learning as a solution to HPO will encounter the sparse reward problem, eventually leading to slow convergence. To address this problem, we employ the intrinsic reward, which introduces the prediction error as the reward signal. Experiments carried on nine tabular datasets and two image classification datasets demonstrate the effectiveness of our model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Data Classification · Metaheuristic Optimization Algorithms Research · Advanced Multi-Objective Optimization Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Hanyang University, Seoul 04763, Korea
Funding
Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
References (38)
Show source-provided references
1. Hyperband: A novel bandit-based approach to hyperparameter optimization
2. AutoML: A survey of the state-of-the-art10.1016/j.knosys.2020.106622 ↗
3. Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 11–14). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487629 ↗
4. Sherpa: Robust hyperparameter optimization for machine learning10.1016/j.softx.2020.100591 ↗
5. Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. (2011, January 12). Algorithms for hyper-parameter optimization. Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain.
6. Random search for hyper-parameter optimization
7. Brochu, E., Cora, V.M., and De Freitas, N. (2010). A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv.
8. Taking the human out of the loop: A review of Bayesian optimization10.1109/JPROC.2015.2494218 ↗
9. Optimization by simulated annealing10.1126/science.220.4598.671 ↗
10. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., and Kurakin, A. (2017, January 6–11). Large-Scale Evolution of Image Classifiers. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia.
11. Jamieson, K., and Talwalkar, A. (2016, January 9–11). Non-stochastic best arm identification and hyperparameter optimization. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Cadiz, Spain.
12. Falkner, S., Klein, A., and Hutter, F. (2018, January 10–15). BOHB: Robust and efficient hyperparameter optimization at scale. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
13. Awad, N., Mallik, N., and Hutter, F. (2021). Dehb: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. arXiv.10.24963/ijcai.2021/296 ↗
14. Differential evolution—A simple and efficient heuristic for global optimization over continuous spaces10.1023/A:1008202821328 ↗
15. Efficient hyperparameter optimization through model-based reinforcement learning10.1016/j.neucom.2020.06.064 ↗
16. Chen, T., and Guestrin, C. (2016, January 13–17). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939785 ↗
17. Isabona, J., Imoize, A.L., and Kim, Y. (2022). Machine Learning-Based Boosted Regression Ensemble Combined with Hyperparameter Tuning for Optimal Adaptive Learning. Sensors, 22.10.3390/s22103776 ↗
18. Hansen, N. (2006). The CMA evolution strategy: A comparing review. Towards a New Evolutionary Computation, Springer.10.1007/3-540-32494-1_4 ↗
19. Liu, C., Wang, H., Liu, N., and Yuan, Z. (2022). Optimizing the Neural Structure and Hyperparameters of Liquid State Machines Based on Evolutionary Membrane Algorithm. Mathematics, 10.10.3390/math10111844 ↗
20. Frank, H., Lars, K., and Joaquin, V. (2019). Hyperparameter optimization. Automated Machine Learning, Springer.
21. Early and robust remaining useful life prediction of supercapacitors using BOHB optimized Deep Belief Network10.1016/j.apenergy.2021.116541 ↗
22. A Markovian decision process
23. Watkins, C.J.C.H. (1989). Learning from Delayed Rewards, King’s College.
24. Q-learning10.1007/BF00992698 ↗
25. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 6–11). Trust region policy optimization. Proceedings of the 2015 32th International Conference on Machine Learning, Lille, France.
26. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
27. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
28. Zoph, B., and Le, Q.V. (2016). Neural architecture search with reinforcement learning. arXiv.
29. Jomaa, H.S., Grabocka, J., and Schmidt-Thieme, L. (2019). Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv.
30. A context-based meta-reinforcement learning approach to efficient hyperparameter optimization10.1016/j.neucom.2021.12.086 ↗
31. Dua, D., and Graff, C. (2022, May 16). UCI Machine Learning Repository. Available online: http://archive.ics.uci.edu/ml.
32. OpenML: Networked science in machine learning
33. LeCun, Y., Cortes, C., and Burges, C.J. (2022, May 16). MNIST Handwritten Digit Database. Available online: http://yann.lecun.com/exdb/mnist.
34. Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv.
35. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
36. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
37. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10–15). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
38. Fujimoto, S., Hoof, H., and Meger, D. (2018, January 10–15). Addressing function approximation error in actor-critic methods. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.