Single-Objective Surrogate Models for Continuous Metaheuristics: An Overview
Source abstract
Abstract
This paper presents a comprehensive overview of single-objective surrogate models for continuous metaheuristics, addressing computationally expensive optimization problems. Metaheuristics typically require numerous objective function evaluations that become impractical with expensive simulations. Surrogate-assisted metaheuristics address this by substituting costly evaluations with lower-cost approximations. We examine three fundamental approaches: regression models that predict exact objective function values, classification models that categorize solutions, and ranking models focusing on relative ordering. We analyze various surrogate types and discuss their strengths and limitations. We discuss structural approaches from global to local models, sample management strategies, and recent advances in ensemble methods and adaptive sampling techniques.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Multi-Objective Optimization Algorithms · Optimal Experimental Design Methods · Optimization and Mathematical Programming
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Computer Science, Warsaw University of Technology, 00-665 Warsaw, Poland
References (100+)
Show source-provided references
1. Glover, F., and Kochenberher, G. (2003). Handbook of Meta-Heuristics, Springer.
2. Talbi, E. (2009). Metaheuristics—From Design to Implementation, Wiley.10.1002/9780470496916 ↗
3. A comprehensive survey of fitness approximation in evolutionary computation10.1007/s00500-003-0328-5 ↗
4. Surrogate-assisted evolutionary computation: Recent advances and future challenges10.1016/j.swevo.2011.05.001 ↗
5. Surrogate Models in Evolutionary Single-Objective Optimization: A New Taxonomy and Experimental Study10.1016/j.ins.2021.03.002 ↗
6. A review of surrogate-assisted evolutionary algorithms for expensive optimization problems10.1016/j.eswa.2022.119495 ↗
7. A survey of surrogate-assisted evolutionary algorithms for expensive optimization10.1007/s41965-024-00165-w ↗
8. Evolutionary Computation for Expensive Optimization: A Survey10.1007/s11633-022-1317-4 ↗
9. Recent advances in surrogate-based optimization10.1016/j.paerosci.2008.11.001 ↗
10. Committee-Based Active Learning for Surrogate-Assisted Particle Swarm Optimization of Expensive Problems10.1109/TCYB.2017.2710978 ↗
11. Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels10.1109/TEVC.2005.859463 ↗
12. A twofold infill criterion-driven heterogeneous ensemble surrogate-assisted evolutionary algorithm for computationally expensive problems10.1016/j.knosys.2021.107747 ↗
13. Efficient global optimization algorithm assisted by multiple surrogate techniques10.1007/s10898-012-9892-5 ↗
14. Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions10.1007/s00158-009-0420-2 ↗
15. A Surrogate-Assisted Multiswarm Optimization Algorithm for High-Dimensional Computationally Expensive Problems10.1109/TCYB.2020.2967553 ↗
16. Faster convergence by means of fitness estimation10.1007/s00500-003-0329-4 ↗
17. A Framework for Evolutionary Optimization with Approximate Fitness Functions10.1109/TEVC.2002.800884 ↗
18. A Gaussian Process Surrogate Model Assisted Evolutionary Algorithm for Medium Scale Expensive Optimization Problems10.1109/TEVC.2013.2248012 ↗
19. Surrogate-assisted differential evolution using manifold learning-based sampling for high- dimensional expensive constrained optimization problems10.1016/j.cja.2024.03.026 ↗
20. Boosting Data-Driven Evolutionary Algorithm with Localized Data Generation10.1109/TEVC.2020.2979740 ↗
21. Surrogate-assisted global sensitivity analysis: An overview10.1007/s00158-019-02413-5 ↗
22. A Classifier-Ensemble-Based Surrogate-Assisted Evolutionary Algorithm for Distributed Data-Driven Optimization10.1109/TEVC.2024.3361000 ↗
23. Le, H.L., Landa-Silva, D., Galar, M., Garcia, S., and Triguero, I. (2020, January 19–24). A Hybrid Surrogate Model for Evolutionary Undersampling in Imbalanced Classification. Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), Glasgow, UK.10.1109/CEC48606.2020.9185774 ↗
24. Multisurrogate-Assisted Ant Colony Optimization for Expensive Optimization Problems with Continuous and Categorical Variables10.1109/TCYB.2021.3064676 ↗
25. Ryan, C., Soule, T., Keijzer, M., Tsang, E., Poli, R., and Costa, E. (2003, January 14–16). Decreasing the Number of Evaluations in Evolutionary Algorithms by Using a Meta-model of the Fitness Function. Proceedings of the Genetic Programming, Essex, UK.10.1007/3-540-36599-0 ↗
26. A New Surrogate-Assisted Interactive Genetic Algorithm with Weighted Semisupervised Learning10.1109/TSMCB.2012.2214382 ↗
27. Lu, X., Tang, K., and Yao, X. (2011, January 5–8). Classification-assisted Differential Evolution for computationally expensive problems. Proceedings of the 2011 IEEE Congress of Evolutionary Computation (CEC), New Orleans, LA, USA.10.1109/CEC.2011.5949859 ↗
28. Boosting evolutionary optimization via fuzzy-classification-assisted selection10.1016/j.ins.2020.01.050 ↗
29. Feasibility Structure Modeling: An Effective Chaperone for Constrained Memetic Algorithms10.1109/TEVC.2009.2039141 ↗
30. Comparative study of regression vs pairwise models for surrogate-based heuristic optimisation10.1016/j.swevo.2022.101176 ↗
31. A Duplication Analysis-Based Evolutionary Algorithm for Biobjective Feature Selection10.1109/TEVC.2020.3016049 ↗
32. A pairwise ranking estimation model for surrogate-assisted evolutionary algorithms10.1007/s40747-023-01113-4 ↗
33. CMA evolution strategy assisted by kriging model and approximate ranking10.1007/s10489-018-1193-3 ↗
34. Schaefer, R., Cotta, C., Kołodziej, J., and Rudolph, G. (2010, January 11–15). Comparison-Based Optimizers Need Comparison-Based Surrogates. Proceedings of the Parallel Problem Solving from Nature, PPSN XI, Krakov, Poland.10.1007/978-3-642-15844-5 ↗
35. Blank, J., and Deb, K. (2021, January 10–14). PSAF: A probabilistic surrogate-assisted framework for single-objective optimization. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’21, Lille, France.10.1145/3449639.3459297 ↗
36. A Multioperator Search Strategy Based on Cheap Surrogate Models for Evolutionary Optimization10.1109/TEVC.2015.2449293 ↗
37. Runarsson, T.P., Beyer, H.G., Burke, E., Merelo-Guervós, J.J., Whitley, L.D., and Yao, X. (2006, January 9–13). Ordinal Regression in Evolutionary Computation. Proceedings of the Parallel Problem Solving from Nature—PPSN IX, Reykjavik, Iceland.10.1007/11844297 ↗
38. A new self-adaptation scheme for differential evolution10.1016/j.neucom.2014.04.071 ↗
39. Runarsson, T. (2004, January 18–22). Constrained Evolutionary Optimization by Approximate Ranking and Surrogate Models. Proceedings of the Parallel Problem Solving from Nature—PPSN VIII, Birmingham, UK.10.1007/978-3-540-30217-9_41 ↗
40. Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer. [2nd ed.].10.1007/978-0-387-84858-7 ↗
41. Myers, R., Montgomery, D., and Anderson-Cook, C. (2009). Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Wiley.
42. Shashaani, S., Hunter, S.R., and Pasupathy, R. (2016, January 11–14). ASTRO-DF: Adaptive sampling trust-region optimization algorithms, heuristics, and numerical experience. Proceedings of the 2016 Winter Simulation Conference, WSC ’16, Washington, DC, USA.10.1109/WSC.2016.7822121 ↗
43. Boyd, S.P., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.10.1017/CBO9780511804441 ↗
44. Tawarmalani, M., and Sahinidis, N. (2002). Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming: Theory, Algorithms, Software, and Applications, Springer. Nonconvex Optimization and Its Applications.10.1007/978-1-4757-3532-1 ↗
45. Powell, M. (2009). The BOBYQA Algorithm for Bound Constrained Optimization without Derivatives, Department of Applied Mathematics and Theoretical Physics. Technical Report.
46. Box, G.E.P., and Wilson, K.B. (1992). On the Experimental Attainment of Optimum Conditions. Breakthroughs in Statistics: Methodology and Distribution, Springer.10.1007/978-1-4612-4380-9_23 ↗
47. Efficient Generalized Surrogate-Assisted Evolutionary Algorithm for High-Dimensional Expensive Problems10.1109/TEVC.2019.2919762 ↗
48. A RBF-based constrained global optimization algorithm for problems with computationally expensive objective and constraints10.1007/s00158-018-1987-2 ↗
49. A Surrogate Assisted Approach for Single-Objective Bilevel Optimization10.1109/TEVC.2017.2670659 ↗
50. Box, G.E.P., and Draper, N.R. (1987). Empirical Model-Building and Response Surfaces, Wiley.
51. Holeňa, M., Linke, D., Rodemerck, U., and Bajer, L. (2010, January 14–16). Neural networks as surrogate models for measurements in optimization algorithms. Proceedings of the 17th International Conference on Analytical and Stochastic Modeling Techniques and Applications, ASMTA’10, Cardiff, UK.10.1007/978-3-642-13568-2_25 ↗
52. Rakitianskaia, A., and Engelbrecht, A. (2015, January 7–10). Measuring Saturation in Neural Networks. Proceedings of the 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, South Africa.10.1109/SSCI.2015.202 ↗
53. Reducing Fitness Evaluations Using Clustering Techniques and Neural Network Ensembles
54. Breiman, L., Friedman, J., Olshen, R.A., and Stone, C.J. (1984). Classification and Regression Trees, Chapman and Hall/CRC. [1st ed.].
55. Zheng, Y., Fu, X., and Xuan, Y. (2019, January 2–4). Data-Driven Optimization Based on Random Forest Surrogate. Proceedings of the 2019 6th International Conference on Systems and Informatics (ICSAI), Shanghai, China.10.1109/ICSAI48974.2019.9010547 ↗
56. Bäck, T., van Stein, N., Wagner, C., Garibaldi, J.M., Marcelloni, F., Lam, H.K., Cottrell, M., Doctor, F., Filipe, J., and Warwick, K. (2025, January 14–16). What Is the Impact of Typical Surrogate Models on the Performance of the JADE Algorithm?. Proceedings of the Computational Intelligence, Kuala Lumpur, Malaysia.10.1007/978-3-031-85252-7 ↗
57. Naharro, P.S., LaTorre, A., and Peña, J.M. (2021, January 10–14). Surrogate-based optimisation for a hospital simulation scenario using pairwise classifiers. Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’21, Lille, France.10.1145/3449726.3463283 ↗
58. Instance-based learning algorithms10.1023/A:1022689900470 ↗
59. Tenne, Y., and Goh, C.K. (2010). A Survey of Fitness Approximation Methods Applied in Evolutionary Algorithms. Computational Intelligence in Expensive Optimization Problems, Springer.10.1007/978-3-642-12775-5 ↗
60. Archetti, F., and Candelieri, A. (2019). Bayesian Optimization and Data Science, Springer.10.1007/978-3-030-24494-1 ↗
61. Wang, X., Jin, Y., Schmitt, S., and Olhofer, M. (2022). Recent Advances in Bayesian Optimization. arXiv.
62. Fonseca, L., Barbosa, H., and Lemonge, A. (2010). On similarity-based surrogate models for expensive single-and multi-objective evolutionary optimization. Computational Intelligence in Expensive Optimization Problems, Springer.10.1007/978-3-642-10701-6_9 ↗
63. Tian, J., Tan, Y., Sun, C., Zeng, J., and Jin, Y. (2016, January 6–9). A self-adaptive similarity-based fitness approximation for evolutionary optimization. Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece.
64. Constraint-handling techniques in surrogate-assisted evolutionary optimization. An empirical study10.1016/j.asoc.2018.08.016 ↗
65. Krawczyk, K., and Arabas, J. (2023, January 15–19). JADE with k Nearest Neighbors Surrogate Model. Proceedings of the Companion Conference on Genetic and Evolutionary Computation, GECCO ’23, Lisbon, Portugal.10.1145/3583133.3596382 ↗
66. A tutorial on support vector regression10.1023/B:STCO.0000035301.49549.88 ↗
67. Santin, G., and Haasdonk, B. (2021). Kernel methods for surrogate modeling. System- and Data-Driven Methods and Algorithms, De Gruyter.10.1515/9783110498967-009 ↗
68. Díaz-Manríquez, A., Toscano-Pulido, G., and Gómez-Flores, W. (2011, January 5–8). On the selection of surrogate models in evolutionary optimization algorithms. Proceedings of the 2011 IEEE Congress of Evolutionary Computation (CEC), New Orleans, LA, USA.10.1109/CEC.2011.5949881 ↗
69. A multi-fidelity surrogate model based on support vector regression10.1007/s00158-020-02522-6 ↗
70. Li, Q., Steven, G., and Zhang, Z. (2015, January 7–12). A Novel Constraint Handling Strategy for Expensive Optimization Problems. Proceedings of the Advances in Structural and Multidisciplinary Optimization, Proceedings of the 11th World Congress of Structural and Multidisciplinary Optimization (WCSMO-11), Sydney, Australia.
71. Low cost PSO using metamodels and inexact pre-evaluation: Application to aerodynamic shape design10.1016/j.cma.2008.11.019 ↗
72. Evolutionary Programming for High-Dimensional Constrained Expensive Black-Box Optimization Using Radial Basis Functions10.1109/TEVC.2013.2262111 ↗
73. On choosing “optimal” shape parameters for RBF approximation10.1007/s11075-007-9072-8 ↗
74. A Three-Level Radial Basis Function Method for Expensive Optimization10.1109/TCYB.2021.3061420 ↗
75. An on-line variable-fidelity surrogate-assisted harmony search algorithm with multi-level screening strategy for expensive engineering design optimization10.1016/j.knosys.2019.01.004 ↗
76. A bi-stage surrogate-assisted hybrid algorithm for expensive optimization problems10.1007/s40747-021-00277-1 ↗
77. Hybrid Surrogate-Based Constrained Optimization with a New Constraint-Handling Method10.1109/TCYB.2020.3031620 ↗
78. Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization10.1080/0305215X.2012.687731 ↗
79. Surrogate-based optimisation using adaptively scaled radial basis functions10.1016/j.asoc.2019.106050 ↗
80. Max-min surrogate-assisted evolutionary algorithm for robust design10.1109/TEVC.2005.859464 ↗
81. SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems10.1016/j.cor.2012.08.022 ↗
82. On efficient global optimization via universal Kriging surrogate models10.1007/s00158-017-1867-1 ↗
83. Gramacy, R.B. (2020). Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences, Chapman and Hall/CRC. [1st ed.].10.1201/9780367815493 ↗
84. Dual Kriging assisted efficient global optimization of expensive problems with evaluation failures10.1016/j.ast.2020.106006 ↗
85. A distance correlation-based Kriging modeling method for high-dimensional problems10.1016/j.knosys.2020.106356 ↗
86. KASRA: A Kriging-based Adaptive Space Reduction Algorithm for global optimization of computationally expensive black-box constrained problems10.1016/j.asoc.2020.106154 ↗
87. SCGOSR: Surrogate-based Constrained Global Optimization using Space Reduction10.1016/j.asoc.2018.01.041 ↗
88. An Improved Differential Evolution Algorithm using Efficient Adapted Surrogate Model for Numerical Optimization10.1016/j.ins.2018.04.024 ↗
89. Exploring and Exploiting a Surrogate10.1002/9780470770801.ch3 ↗
90. Gaussian Processes in Machine Learning
91. Reliable Extrapolation of Deep Neural Operators Informed by Physics or Sparse Observations10.1016/j.cma.2023.116064 ↗
92. A Survey of Adaptive Sampling for Global Metamodeling in Support of Simulation-based Complex Engineering Design10.1007/s00158-017-1739-8 ↗
93. Extrapolative Bayesian Optimization with Gaussian Process and Neural Network Ensemble Surrogate Models10.1002/aisy.202100101 ↗
94. Advances in surrogate based modeling, feasibility analysis and and optimization: A review10.1016/j.compchemeng.2017.09.017 ↗
95. Combining global and local surrogate models to accelerate evolutionary optimization10.1109/TSMCC.2005.855506 ↗
96. Ghassemi, P., Behjat, A., Zeng, C., Lulekar, S.S., Rai, R., and Chowdhury, S. (2020, January 15–19). Physics-Aware Surrogate-Based Optimization with Transfer Mapping Gaussian Processes: For Bio-Inspired Flow Tailoring. Proceedings of the AIAA Aviation 2020 Forum, American Institute of Aeronautics and Astronautics, Virtual Online.10.2514/6.2020-3183 ↗
97. Koziel, S., and Leifsson, L. (2016). Physics-Based Surrogate Modeling Using Response Correction. Simulation-Driven Design by Knowledge-Based Response Correction Techniques, Springer International Publishing.10.1007/978-3-319-30115-0 ↗
98. Ensemble Learning Through Evolutionary Multitasking: A Formulation and Case Study10.1109/TETCI.2024.3369949 ↗
99. Iuliano, E., and Pérez, E.A. (2016). Adaptive Sampling Strategies for Surrogate-Based Aerodynamic Optimization. Application of Surrogate-based Global Optimization to Aerodynamic Design, Springer International Publishing.10.1007/978-3-319-21506-8 ↗
100. Liu, W., Yuen, S.Y., and Sung, C.W. (July, January 30). A Hybrid CMAES Method with Convex Hull Surrogate Model. Proceedings of the 2024 IEEE Congress on Evolutionary Computation (CEC), Yokohama, Japan.10.1109/CEC60901.2024.10612122 ↗