A Literature Survey on Offline Automatic Algorithm Configuration
Source abstract
Abstract
Metaheuristic and heuristic methods have many tunable parameters, and choosing their values can increase their ability to deal with hard optimization problems. Automated approaches for finding good parameter settings have attracted significant research and development efforts in the last few years. Because parameter tuning became commonly utilized in industry and research and there is a significant advancement in this area, a comprehensive review is an important requirement. Although there is very wide literature about algorithm configuration problems, a detailed survey analysis has not been conducted yet to the best of our knowledge. In this paper, we will briefly explain the automatic algorithm configuration problem and then survey the automated methods developed to handle this problem. After explaining the logic of these methods, we also argued about their main advantages and disadvantages to help researchers or practitioners select the best possible method for their specific problem. Moreover, some recommendations and possible future directions for this topic are provided as a conclusion.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Multi-Objective Optimization Algorithms · Metaheuristic Optimization Algorithms Research · Industrial Vision Systems and Defect Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Industrial Engineering, Gaziantep University, Gaziantep 27010, Turkey
References (100+)
Show source-provided references
1. Performance evaluation of automatically tuned continuous optimizers on different benchmark sets10.1016/j.asoc.2014.11.006 ↗
2. Birattari, M., and Kacprzyk, J. (2009). Tuning Metaheuristics: A Machine Learning Perspective, Springer.10.1007/978-3-642-00483-4 ↗
3. Efficient and robust parameter tuning for heuristic algorithms
4. Parameter Tuning for configuring and analyzing evolutionary algorithms10.1016/j.swevo.2011.02.001 ↗
5. Eiben, A.E., and Smit, S.K. (2011). Evolutionary Algorithm Parameters and Methods to Tune Them. Autonomous Search, Springer.10.1007/978-3-642-21434-9_2 ↗
6. Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2011, January 17–21). Sequential Model-Based Optimization for General Algorithm Configuration. Proceedings of the International Conference on Learning and Intelligent Optimization, Rome, Italy.10.1007/978-3-642-25566-3_40 ↗
7. The irace package: Iterated racing for automatic algorithm configuration
8. Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2012). Parallel Algorithm Configuration. Learning and Intelligent Optimization, Springer.10.1007/978-3-642-34413-8_5 ↗
9. Ansótegui, C., Sellmann, M., and Tierney, K. (2009). A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms. International Conference on Principles and Practice of Constraint Programming, Springer.10.1007/978-3-642-04244-7_14 ↗
10. Bezerra, L.C.T., López-Ibánez, M., and Stützle, T. (2017). Automatic Configuration of Multi-objective Optimizers and Multi-Objective Configuration, Université Libre de Bruxelles. Technical Report TR/IRIDIA/2017-011, IRIDIA.
11. Hoos, H.H. (2011). Automated Algorithm Configuration and Parameter Tuning. Autonomous Search, Springer.10.1007/978-3-642-21434-9_3 ↗
12. A beginner’s guide to tuning methods10.1016/j.asoc.2013.12.017 ↗
13. A survey of automatic parameter tuning methods for metaheuristics10.1109/TEVC.2019.2921598 ↗
14. Schede, E., Brandt, J., Tornede, A., Wever, M., Bengs, V., Hüllermeier, E., and Tierney, K. (2022). A Survey of Methods for Automated Algorithm Configuration. arXiv.10.24963/ijcai.2023/791 ↗
15. Gratch, J., and Dejong, G. (1992). COMPOSER: A Probabilistic Solution to the Utility Problem in Speed-Up Learning, RIC.
16. Automatically configuring constraint satisfaction programs: A case study10.1007/BF00143877 ↗
17. Adaptive search using a reproductive meta-plan10.1108/eb005486 ↗
18. Optimization of control parameters for genetic algorithms10.1109/TSMC.1986.289288 ↗
19. De Landgraaf, W.A., Eiben, A.E., and Nannen, V. (2007, January 25–28). Parameter Calibration Using Meta-Algorithms. Proceedings of the IEEE Congress on Evolutionary Computation, Singapore.10.1109/CEC.2007.4424456 ↗
20. ISAC-Instance-Specific Algorithm Configuration
21. MaxSAT by improved instance-specific algorithm configuration10.1016/j.artint.2015.12.006 ↗
22. Malitsky, Y., Mehta, D., O’Sullivan, B., and Simonis, H. (2013). Tuning Parameters of Large Neighborhood Search for the Machine Reassignment Problem. International Conference on AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, Springer.10.1007/978-3-642-38171-3_12 ↗
23. Ansótegui, C., Malitsky, Y., Samulowitz, H., Sellmann, M., and Tierney, K. (2015, January 25–31). Model-Based Genetic Algorithms for Algorithm Configuration. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
24. Ansótegui, C., Pon, J., Sellmann, M., and Tierney, K. (2021, January 5–9). PyDGGA: Distributed GGA for Automatic Configuration. Proceedings of the International Conference on Theory and Applications of Satisfiability Testing, Barcelona, Spain.10.1007/978-3-030-80223-3_2 ↗
25. Nannen, V., and Eiben, A.E. (2006, January 8–12). A method for parameter calibration and relevance estimation in evolutionary algorithms. Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, Washington, DC, USA.10.1145/1143997.1144029 ↗
26. Nannen, V., and Eiben, A.E. (2007, January 25–28). Efficient Relevance Estimation and Value Calibration of Evolutionary Algorithm Parameters. Proceedings of the IEEE Congress on Evolutionary Computation, Singapore.10.1109/CEC.2007.4424460 ↗
27. A survey of optimization by building and using probabilistic models10.1023/A:1013500812258 ↗
28. Smit, S.K., and Eiben, A.E. (2010). Using Entropy for Parameter Analysis of Evolutionary Algorithms. Experimental Methods for the Analysis of Optimization Algorithms, Springer.10.1007/978-3-642-02538-9_12 ↗
29. Nannen, V., Smit, S.K., and Eiben, A.E. (2008, January 5–9). Costs and Benefits of Tuning Parameters of Evolutionary Algorithms. Proceedings of the International Conference on Parallel Problem Solving from Nature, Leiden, The Netherlands.10.1007/978-3-540-87700-4_53 ↗
30. Smit, S.K., and Eiben, A.E. (2010, January 18–23). Beating the ‘World Champion’ Evolutionary Algorithm via REVAC Tuning. Proceedings of the IEEE Congress on Evolutionary Computation (CEC), Barcelona, Spain.10.1109/CEC.2010.5586026 ↗
31. Smit, S.K., and Eiben, A.E. (2010, January 20–22). Parameter Tuning of Evolutionary Algorithms: Generalist vs. Specialist. Proceedings of the European Conference on the Applications of Evolutionary Computation, Madrid, Spain.10.1007/978-3-642-12239-2_56 ↗
32. No free lunch theorems for optimization10.1109/4235.585893 ↗
33. Smit, S.K. (2022, June 10). MOBAT. Available online: http://mobat.sourceforge.net.
34. Zhang, T., Georgiopoulos, M., and Anagnostopoulos, G.C. (2013, January 6–10). S-Race: A multi-Objective Racing Algorithm. Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, Amsterdam, The Netherlands.10.1145/2463372.2463561 ↗
35. Zhang, T., Georgiopoulos, M., and Anagnostopoulos, G.C. (2015, January 11–15). SPRINT Multi-Objective Model Racing. Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, Madrid, Spain.10.1145/2739480.2754791 ↗
36. Multi-objective model selection via racing10.1109/TCYB.2015.2456187 ↗
37. Blot, A., Hoos, H.H., Jourdan, L., Kessaci-Marmion, M.É., and Trautmann, H. (June, January 29). MO-ParamILS: A multi-objective automatic algorithm configuration framework. Proceedings of the International Conference on Learning and Intelligent Optimization, Ischia, Italy.
38. Smit, S.K., Eiben, A.E., and Szlávik, Z. (2010, January 24–26). An MOEA-based Method to Tune EA Parameters on Multiple Objective Functions. Proceedings of the International Conference on Evolutionary Computation Theory and Applications, Valencia, Spain.
39. Dréo, J. (2009, January 8–12). Using Performance Fronts for Parameter Setting of Stochastic Metaheuristics. Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, Montréal, Canada.10.1145/1570256.1570301 ↗
40. Smit, S.K., and Eiben, A.E. (2011). Multi-Problem Parameter Tuning using Bonesa. Artificial Evolution, Springer.
41. Ugolotti, R., and Cagnoni, S. (2014, January 18–20). Analysis of Evolutionary Algorithms Using Multi-Objective Parameter Tuning. Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, Nanchang, China.10.1145/2576768.2598226 ↗
42. A fast and elitist multiobjective genetic algorithm: NSGA-II10.1109/4235.996017 ↗
43. Ugolotti, R., Sani, L., and Cagnoni, S. (2019). What Can We Learn from Multi-Objective Meta-Optimization of Evolutionary Algorithms in Continuous Domains?. Mathematics, 7.10.3390/math7030232 ↗
44. Tuning optimization algorithms under multiple objective function evaluation budgets10.1109/TEVC.2014.2322883 ↗
45. Mota: A many-objective tuning algorithm specialized for tuning under multiple objective function evaluation budgets10.1162/EVCO_a_00163 ↗
46. Hutter, F., Hoos, H.H., and Stützle, T. (2007). Automatic Algorithm Configuration Based on Local Search, AAAI.
47. ParamILS: An automatic algorithm configuration framework10.1613/jair.2861 ↗
48. Fine-tuning of algorithms using fractional experimental designs and local search10.1287/opre.1050.0243 ↗
49. Exploring variable neighborhood search for automatic algorithm configuration10.1016/j.endm.2017.03.022 ↗
50. Blot, A., Pernet, A., Jourdan, L., Kessaci-Marmion, M.É., and Hoos, H.H. (2017, January 10–13). Automatically Configuring Multi-Objective Local Search using Multi-Objective Optimization. Proceedings of the International Conference on Evolutionary Multi-Criterion Optimization, Shenzhen, China.10.1007/978-3-319-54157-0_5 ↗
51. Automatic Configuration of Multi-Objective Local Search Algorithms for Permutation Problems10.1162/evco_a_00240 ↗
52. Blot, A., Hoos, H.H., Kessaci, M.É., and Jourdan, L. (2018, January 5–7). Automatic Configuration of Bi-Objective Optimization Algorithms: Impact of Correlation Between Objectives. Proceedings of the IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), Volos, Greece.10.1109/ICTAI.2018.00093 ↗
53. Hutter, F., Babic, D., Hoos, H.H., and Hu, A.J. (2007). Boosting Verification by Automatic Tuning of Decision Procedures. Formal Methods in Computer-Aided Design, Springer.10.1109/FMCAD.2007.4401979 ↗
54. Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2010, January 14–18). Automated Configuration of Mixed Integer Programming Solvers. Proceedings of the International Conference on Integration of Artificial Intelligence (AI) and Operations Research (OR) Techniques in Constraint Programming, Bologna, Italy.10.1007/978-3-642-13520-0_23 ↗
55. Parameter tuning with Chess Rating System (CRS-Tuning) for meta-heuristic algorithms10.1016/j.ins.2016.08.066 ↗
56. The configurable SAT solver challenge (CSSC)10.1016/j.artint.2016.09.006 ↗
57. Birattari, M., Stützle, T., Paquete, L., and Varrentrapp, K. (2002, January 9–13). A Racing Algorithm for Configuring Metaheuristics. Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, New York, NY, USA.
58. Maron, O., and Moore, A.W. (1994). Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation. Advances in Neural Information Processing Systems, MIT Press.
59. Moore, A.W., and Lee, M.S. (1994). Efficient Algorithms for Minimizing Cross-Validation Error. Machine Learning Proceedings, Elsevier.10.1016/B978-1-55860-335-6.50031-3 ↗
60. Conover, W.J. (1999). Practical Nonparametric Statistics, John Wiley and Sons.
61. Balaprakash, P., Birattari, M., and Stützle, T. (2007). Improvement Strategies for the F-Race Algorithm: Sampling Design and Iterative Refinement. International Workshop on Hybrid Metaheuristics, Springer.10.1007/978-3-540-75514-2_9 ↗
62. Birattari, M., Yuan, Z., Balaprakash, P., and Stützle, T. (2010). F-Race and Iterated F-Race: An Overview. Experimental Methods for the Analysis of Optimization Algorithms, Springer.10.1007/978-3-642-02538-9_13 ↗
63. An automatic multi-objective evolutionary algorithm for the hybrid flowshop scheduling problem with consistent sublots10.1016/j.knosys.2021.107819 ↗
64. Cáceres, L.P., López-Ibáñez, M., Hoos, H., and Stützle, T. (2017, January 19–21). An experimental Study of adaptive capping in irace. Proceedings of the International Conference on Learning and Intelligent Optimization, Nizhny Novgorod, Russia.10.1007/978-3-319-69404-7_17 ↗
65. Cáceres, L.P., Bischl, B., and Stützle, T. (2017, January 15–19). Evaluating random forest models for irace. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Berlin, Germany.10.1145/3067695.3082057 ↗
66. Improving the performance of metaheuristics: An approach combining response surface methodology and racing algorithms10.1155/2015/167031 ↗
67. Eriksson, L., Johansson, E., Kettaneh-Wold, N., Wikström, C., and Wold, S. (2000). Design of Experiments, Principles and Applications, Learn ways AB.
68. Ridge, E. (2007). Design of Experiments for the Tuning of Optimization Algorithms, Citeseer.
69. Ridge, E., and Kudenko, D. (2006, January 9). Sequential Experiment Designs for Screening and Tuning Parameters of Stochastic Heuristics. Proceedings of the Workshop on Empirical Methods for the Analysis of Algorithms at the Ninth International Conference on Parallel Problem Solving from Nature (PPSN), Reykjavik, Iceland.
70. Ridge, E., and Kudenko, D. (2007). Tuning the Performance of the MMAS Heuristic. International Workshop on Engineering Stochastic Local Search Algorithms, Springer.10.1007/978-3-540-74446-7_4 ↗
71. Ridge, E., and Kudenko, D. (2008). Determining Whether a Problem Characteristic Affects Heuristic Performance. Recent Advances in Evolutionary Computation for Combinatorial Optimization, Springer.10.1007/978-3-540-70807-0_2 ↗
72. Ridge, E., and Kudenko, D. (2010). Tuning an Algorithm Using Design of Experiments. Experimental Methods for the Analysis of Optimization Algorithms, Springer.10.1007/978-3-642-02538-9_11 ↗
73. A parameter tuning methodology for metaheuristics based on design of experiments
74. A systematic procedure for setting parameters in simulated annealing algorithms10.1016/S0305-0548(97)00054-3 ↗
75. Using experimental design to find effective parameter settings for heuristics10.1023/A:1026569813391 ↗
76. Dobslaw, F. (2010, January 10–12). A parameter Tuning Framework for Metaheuristics Based on Design of Experiments and Artificial Neural Networks. Proceedings of the International Conference on Computer Mathematics and Natural Computing, Yantai, China.
77. Using fuzzy logic to tune an evolutionary algorithm for dynamic optimization of chemical processes10.1016/j.compchemeng.2011.08.003 ↗
78. Gunawan, A., and Lau, H.C. (2011, January 7–21). Fine-Tuning Algorithm Parameters Using the Design of Experiments Approach. Proceedings of the International Conference on Learning and Intelligent Optimization, Rome, Italy.10.1007/978-3-642-25566-3_21 ↗
79. Gunawan, A., Lau, H.C., and Wong, E. (2013). Real-World Parameter Tuning Using Factorial Design with Parameter Decomposition. Advances in Metaheuristics, Springer.10.1007/978-1-4614-6322-1_3 ↗
80. Montgomery, D.C. (2017). Design and Analysis of Experiments, John Wiley and Sons.
81. Efficient global optimization of expensive black-box functions10.1023/A:1008306431147 ↗
82. Design and analysis of computer experiments
83. Global optimization of stochastic black-box systems via sequential kriging meta-models10.1007/s10898-005-2454-3 ↗
84. Sequential design of computer experiments to minimize integrated response functions
85. Bartz-Beielstein, T., Lasarczyk, C.W., and Preus, M. (2005, January 2–5). Sequential Parameter Optimization. Proceedings of the IEEE Congress on Evolutionary Computation, Edinburgh, UK.
86. Hutter, F., Hoos, H.H., Leyton-Brown, K., and Murphy, K.P. (2009, January 8–12). An Experimental Investigation of Model-Based Parameter Optimization: SPO and Beyond. Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, Montreal, QC, Canada.10.1145/1569901.1569940 ↗
87. Bartz-Beielstein, T., and Preuss, M. (2006, January 9). Considerations of Budget Allocation for Sequential Parameter Optimization (SPO). Proceedings of the Workshop on Empirical Methods for the Analysis of Algorithms, Reykjavik, Iceland.
88. Lasarczyk, C.W. (2007). Genetische Programmierung Einer Algorithmischen Chemie. [Ph.D. Thesis, Technische Universität Dortmund].
89. Chen, J., Chen, C., and Kelton, D. (2005, January 06). Optimal Computing Budget Allocation of Indifference-Zone-Selection Procedures. Working Paper. Available online: http://www.cba.uc.edu/faculty/keltonwd.
90. (2022, June 10). Available online: https://cran.r-project.org/web/packages/SPOT/index.html.
91. Preuss, M., and Bartz-Beielstein, T. (2007). Sequential Parameter Optimization Applied to Self-Adaptation for Binary-Coded Evolutionary Algorithms. Parameter Setting in Evolutionary Algorithms, Springer.10.1007/978-3-540-69432-8_5 ↗
92. Konen, W., Koch, P., Flasch, O., and Bartz-Beielstein, T. (2010). Parameter-Tuned Data Mining: A general Framework, Cologne University of Applied Sciences.10.1145/2001576.2001844 ↗
93. Konen, W., Koch, P., Flasch, O., Bartz-Beielstein, T., and Friese, M. (2011, January 12–16). Tuned Data Mining: A benchmark Study on Different Tuners. Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, Dublin, Ireland.10.1145/2001576.2001844 ↗
94. Hutter, F., Hoos, H.H., Leyton-Brown, K., and Murphy, K. (2010, January 18–22). Time-Bounded Sequential Parameter Optimization. Proceedings of the International Conference on Learning and Intelligent Optimization, Venice, Italy.10.1007/978-3-642-13800-3_30 ↗
95. Hutter, F., Hoos, H., and Leyton-Brown, K. (2013, January 6–10). An Evaluation of Sequential Model-Based Optimization for Expensive Black-Box Functions. Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation, Amsterdam, The Netherlands.10.1145/2464576.2501592 ↗
96. Tanabe, R., and Fukunaga, A. (2015, January 25–28). Tuning Differential Evolution for Cheap, Medium, and Expensive Computational Budgets. Proceedings of the 2015 IEEE Congress on Evolutionary Computation (CEC), Sendai, Japan.10.1109/CEC.2015.7257133 ↗
97. Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 11–14). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487629 ↗
98. Schwarz, H., Kotthoff, L., Hoos, H., Fichtner, W., and Bertsch, V. (2017). Using Automated Algorithm Configuration to Improve the Optimization of Decentralized Energy Systems Modeled as Large-Scale, Two-Stage Stochastic Programs. Working Paper Series in Production and Energy, Karlsruhe Institute of Technology (KIT).
99. Mu, Z., Hoos, H.H., and Stützle, T. (June, January 29). The Impact of Automated Algorithm Configuration on the Scaling Behavior of State-Of-The-Art Inexact TSP Solvers. Proceedings of the International Conference on Learning and Intelligent Optimization, Ischia, Italy.
100. Autofolio: An automatically configured algorithm selector10.1613/jair.4726 ↗