journal article·2025-07-30Open access

A Deep Learning-Driven Black-Box Benchmark Generation Method via Exploratory Landscape Analysis

Haoming Liang, Fuqing Zhao, Tianpeng Xu, Jianlin Zhang

Applied Sciences· Volume 15· Issue 15

Abstract

In the context of algorithm selection, the careful design of benchmark functions and problem instances plays a pivotal role in evaluating the performance of optimization methods. Traditional benchmark functions have been criticized for their limited resemblance to real-world problems and insufficient coverage of the problem space. Exploratory landscape analysis (ELA) offers a systematic framework for characterizing objective functions, based on quantitative landscape features. This study proposes a method for generating benchmark functions tailored to single-objective continuous optimization problems with boundary constraints using predefined ELA feature vectors to guide their construction. The process begins with the creation of random decision variables and corresponding objective values, which are iteratively adjusted using the covariance matrix adaptation evolution strategy (CMA-ES) to ensure alignment with a target ELA feature vector within a specified tolerance. Once the feature criteria are met, the resulting topological map point is used to train a neural network to produce a surrogate function that retains the desired landscape characteristics. To validate the proposed approach, functions from the well-known Black Box Optimization Benchmark (BBOB) suite are replicated, and novel functions are generated with unique ELA feature combinations not found in the original suite. The experiment results demonstrate that the synthesized landscapes closely resemble their BBOB counterparts and preserve the consistency of the algorithm rankings, thereby supporting the effectiveness of the proposed approach.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Multi-Objective Optimization Algorithms · Metaheuristic Optimization Algorithms Research · Ship Hydrodynamics and Maneuverability

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Computer and Communication, Lanzhou University of Technology, Lanzhou 730050, China

Funding

National Natural Science Foundation of China · Key Program of National Natural Science Foundation of Gansu Province · Industry Support Project of Gansu Province College

References (45)

Show source-provided references
1. Lang, R.D., and Engelbrecht, A.P. (2021). An exploratory landscape analysis-based benchmark suite. Algorithms, 14.10.3390/a14030078 ↗
2. Benchmark functions for the CEC 2013 special session and competition on large-scale global optimization
3. Hansen, N., Finck, S., Ros, R., and Auger, A. (2009). Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions, INRIA.
4. Rapin, J., and Teytaud, O. (2018). Nevergrad—A Gradient-Free Optimization Platform, GitHub.
5. Long, F.X., van Stein, B., Frenzel, M., Krause, P., Gitterle, M., and Bäck, T. (2022, January 9–13). Learning the characteristics of engineering optimization problems with applications in automotive crash. Proceedings of the Genetic and Evolutionary Computation Conference, Boston, MA, USA.10.1145/3512290.3528712 ↗
6. Prager, R.P., Dietrich, K., Schneider, L., Schäpermeier, L., Bischl, B., Kerschke, P., Trautmann, H., and Mersmann, O. (September, January 30). Neural networks as black-box benchmark functions optimized for exploratory landscape features. Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms, Potsdam, Germany.
7. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning10.1162/evco_a_00236 ↗
8. Kostovska, A., Vermetten, D., Džeroski, S., Doerr, C., Korosec, P., and Eftimov, T. (2022, January 9–13). The importance of landscape features for performance prediction of modular CMA-ES variants. Proceedings of the Genetic and Evolutionary Computation Conference, Boston, MA, USA.10.1145/3512290.3528832 ↗
9. Prager, R.P., Seiler, M.V., Trautmann, H., and Kerschke, P. (2022, January 14–18). Automated algorithm selection in single-objective continuous optimization: A comparative study of deep learning and landscape analysis methods. Proceedings of the International Conference on Parallel Problem Solving from Nature, Hagenberg, Austria.10.1007/978-3-031-14714-2_1 ↗
10. Generating new space-filling test instances for continuous black-box optimization10.1162/evco_a_00262 ↗
11. A general-purpose tunable landscape generator10.1109/TEVC.2005.863628 ↗
12. A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem10.1007/s10472-013-9341-2 ↗
13. Malan, K.M. (2021). A survey of advances in landscape analysis for optimisation. Algorithms, 14.10.3390/a14020040 ↗
14. Garden, R.W., and Engelbrecht, A.P. (2014, January 6–11). Analysis and classification of optimisation benchmark functions and benchmark suites. Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC), Beijing, China.10.1109/CEC.2014.6900240 ↗
15. A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis10.1016/j.swevo.2024.101669 ↗
16. Christie, L.A., Brownlee, A.E., and Woodward, J.R. (2018, January 15–19). Investigating benchmark correlations when comparing algorithms with parameter tuning. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Kyoto, Japan.10.1145/3205651.3205747 ↗
17. Dietrich, K., and Mersmann, O. (2022, January 10–14). Increasing the diversity of benchmark function sets through affine recombination. Proceedings of the International Conference on Parallel Problem Solving from Nature, Dortmund, Germany.10.1007/978-3-031-14714-2_41 ↗
18. Jankovic, A., Eftimov, T., and Doerr, C. (2021). Towards feature-based performance regression using trajectory data. Proceedings of the Applications of Evolutionary Computation: 24th International Conference, EvoApplications 2021, Held as Part of EvoStar 2021, Virtual Event, April 7–9, 2021, Proceedings 24, Springer.
19. Malan, K.M., and Engelbrecht, A.P. (2009, January 18–21). Quantifying ruggedness of continuous landscapes using entropy. Proceedings of the 2009 IEEE Congress on Evolutionary Computation, Trondheim, Norway.10.1109/CEC.2009.4983112 ↗
20. Automated algorithm selection: Survey and perspectives10.1162/evco_a_00242 ↗
21. Renau, Q., Doerr, C., Dreo, J., and Doerr, B. (2020, January 5–9). Exploratory landscape analysis is strongly sensitive to the sampling strategy. Proceedings of the Parallel Problem Solving from Nature–PPSN XVI: 16th International Conference, PPSN 2020, Leiden, The Netherlands. Proceedings, Part II 16.10.1007/978-3-030-58115-2_10 ↗
22. Renau, Q., Dreo, J., Doerr, C., and Doerr, B. (2021). Towards explainable exploratory landscape analysis: Extreme feature selection for classifying BBOB functions. Proceedings of the Applications of Evolutionary Computation: 24th International Conference, EvoApplications 2021, Held as Part of EvoStar 2021, Virtual Event, April 7–9, 2021, Proceedings 24, Springer.
23. Prager, R.P., and Trautmann, H. (2023, January 12–14). Nullifying the inherent bias of non-invariant exploratory landscape analysis features. Proceedings of the International Conference on the Applications of Evolutionary Computation (Part of EvoStar), Brno, Czech Republic.10.1007/978-3-031-30229-9_27 ↗
24. Derbel, B., Liefooghe, A., Vérel, S., Aguirre, H., and Tanaka, K. (2019, January 27–29). New features for continuous exploratory landscape analysis based on the SOO tree. Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms, Potsdam, Germany.10.1145/3299904.3340308 ↗
25. Seiler, M.V., Prager, R.P., Kerschke, P., and Trautmann, H. (2022, January 9–13). A collection of deep learning-based feature-free approaches for characterizing single-objective continuous fitness landscapes. Proceedings of the Genetic and Evolutionary Computation Conference, Boston, MA, USA.10.1145/3512290.3528834 ↗
26. Saini, B.S., López-Ibáñez, M., and Miettinen, K. (2019, January 13–17). Automatic surrogate modelling technique selection based on features of optimization problems. Proceedings of the Genetic and Evolutionary Computation Conference Companion, Prague, Czech Republic.10.1145/3319619.3326890 ↗
27. Pflacco: Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems in Python10.1162/evco_a_00341 ↗
28. Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011, January 12–16). Exploratory landscape analysis. Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, Dublin, Ireland.10.1145/2001576.2001690 ↗
29. Kerschke, P., Preuss, M., Wessing, S., and Trautmann, H. (2015, January 11–15). Detecting funnel structures by means of exploratory landscape analysis. Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, Madrid, Spain.10.1145/2739480.2754642 ↗
30. Hansen, N. (2009, January 8–12). Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed. Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, Montréal, QC, Canada.10.1145/1570256.1570333 ↗
31. Pytorch: An imperative style, high-performance deep learning library
32. A review of comparative studies of spatial interpolation methods in environmental sciences: Performance and impact factors10.1016/j.ecoinf.2010.12.003 ↗
33. Emmendorfer, L.R., and Dimuro, G.P. (2020, January 3–5). A novel formulation for inverse distance weighting from weighted linear regression. Proceedings of the Computational Science–ICCS 2020: 20th International Conference, Amsterdam, The Netherlands. Proceedings, Part II 20.
34. Deep-ELA: Deep Exploratory Landscape Analysis with Self-Supervised Pretrained Transformers for Single-Objective and Multiobjective Continuous Optimization Problems10.1162/evco_a_00367 ↗
35. Interpolation, approximation, and controllability of deep neural networks10.1137/23M1599744 ↗
36. Bansal, J.C. (2019). Particle swarm optimization. Evolutionary and Swarm Intelligence Algorithms, Springer.10.1007/978-3-319-91341-4 ↗
37. An expectation–maximization algorithm for the matrix normal distribution with an application in remote sensing10.1016/j.jmva.2018.03.010 ↗
38. Krejca, M.S., and Witt, C. (2019). Theory of estimation-of-distribution algorithms. Theory of Evolutionary Computation: Recent Developments in Discrete Optimization, Springer.10.1007/978-3-030-29414-4_9 ↗
39. Convergence of the Nelder-Mead method10.1007/s11075-021-01221-7 ↗
40. Diagonal acceleration for covariance matrix adaptation evolution strategies10.1162/evco_a_00260 ↗
41. Li, L., and Talwalkar, A. (2020, January 3–6). Random search and reproducibility for neural architecture search. Proceedings of the Uncertainty in Artificial Intelligence, Toronto, ON, Canada.
42. On the construction of linear approximations of line flow constraints for AC optimal power flow10.1109/TPWRS.2018.2874173 ↗
43. Differential Evolution: A survey of theoretical analyses10.1016/j.swevo.2018.06.010 ↗
44. Autorank: A python package for automated ranking of classifiers10.21105/joss.02173 ↗
45. Rodríguez, A.P., and Melgarejo, M.A. (2024, January 23–24). Optimization Landscape Classification with Multi-Scale Sampling and Deep Neural Networks. Proceedings of the 2024 Sixth International Conference on Intelligent Computing in Data Sciences (ICDS), Marrakech, Morocco.10.1109/ICDS62089.2024.10756341 ↗