Benchmarking the performance of Bayesian optimization across multiple experimental materials science domains
Source abstract
Abstract
Abstract Bayesian optimization (BO) has been leveraged for guiding autonomous and high-throughput experiments in materials science. However, few have evaluated the efficiency of BO across a broad range of experimental materials domains. In this work, we quantify the performance of BO with a collection of surrogate model and acquisition function pairs across five diverse experimental materials systems. By defining acceleration and enhancement metrics for materials optimization objectives, we find that surrogate models such as Gaussian Process (GP) with anisotropic kernels and Random Forest (RF) have comparable performance in BO, and both outperform the commonly used GP with isotropic kernels. GP with anisotropic kernels has demonstrated the most robustness, yet RF is a close alternative and warrants more consideration because it is free from distribution assumptions, has smaller time complexity, and requires less effort in initial hyperparameter selection. We also raise awareness about the benefits of using GP with anisotropic kernels in future materials optimization campaigns.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Algorithms · Machine Learning in Materials Science · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (55)
Show source-provided references
1. Settles, B. Active learning literature survey (University of Wisconsin-Madison Department of Computer Sciences, 2009).
2. Cohn, D. A., Ghahramani, Z. & Jordan, M. I. Active learning with statistical models. J. Artif. Intell. Res 4, 129–145 (1996).10.1613/jair.295 ↗
3. Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & De Freitas, N. Taking the human out of the loop: a review of bayesian optimization. Proc. IEEE 104, 148–175 (2015).10.1109/JPROC.2015.2494218 ↗
4. Rasmussen, C. E. & Nickisch, H. Gaussian processes for machine learning (gpml) toolbox. J. Mach. Learn. Res. 11, 3011–3015 (2010).
5. Frazier, P. I. A tutorial on bayesian optimization. arXiv Preprint at https://arxiv.org/abs/1807.02811 (2018).
6. Springenberg, J. T., Klein, A., Falkner, S. & Hutter, F. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29, 4134–4142 (2016).
7. Brochu, E., Cora, V. M. & De Freitas, N. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv Preprint at https://arxiv.org/abs/1012.2599 (2010).
8. Frazier, P. I. & Wang, J. Bayesian optimization for materials design. In Information Science for Materials Discovery and Design, 45–75 (Springer, 2016).10.1007/978-3-319-23871-5_3 ↗
9. Eriksson, D., Pearce, M., Gardner, J. R., Turner, R. & Poloczek, M. Scalable global optimization via local bayesian optimization. In Advances in Neural Information Processing Systems 32, 5496–5507 (2019).
10. Wang, Z., Li, C., Jegelka, S. & Kohli, P. Batched high-dimensional bayesian optimization via structural kernel learning. In Int. J. Mach. Learn., 3656–3664 (PMLR, 2017).
11. Solomou, A. et al. Multi-objective bayesian materials discovery: application on the discovery of precipitation strengthened niti shape memory alloys through micromechanical modeling. Mater. Des. 160, 810–827 (2018).10.1016/j.matdes.2018.10.014 ↗
12. Yamawaki, M., Ohnishi, M., Ju, S. & Shiomi, J. Multifunctional structural design of graphene thermoelectrics by bayesian optimization. Sci. Adv. 4, eaar4192 (2018).10.1126/sciadv.aar4192 ↗
13. Bassman, L. et al. Active learning for accelerated design of layered materials. npj Comput. Mater. 4, 1–9 (2018).10.1038/s41524-018-0129-0 ↗
14. Rouet-Leduc, B., Barros, K., Lookman, T. & Humphreys, C. J. Optimisation of gan leds and the reduction of efficiency droop using active machine learning. Sci. Rep. 6, 1–6 (2016).10.1038/srep24862 ↗
15. Xue, D. et al. Accelerated search for batio3-based piezoelectrics with vertical morphotropic phase boundary using bayesian learning. Proc. Natl. Acad. Sci. 113, 13301–13306 (2016).10.1073/pnas.1607412113 ↗
16. Chang, J. et al. Efficient closed-loop maximization of carbon nanotube growth rate using bayesian optimization. Sci. Rep. 10, 1–9 (2020).10.1038/s41598-019-56847-4 ↗
17. MacLeod, B. P. et al. Self-driving laboratory for accelerated discovery of thin-film materials. Sci. Adv. 6, eaaz8867 (2020).10.1126/sciadv.aaz8867 ↗
18. Eyke, N. S., Koscher, B. A. & Jensen, K. F. Toward machine learning-enhanced high-throughput experimentation. Trends Chem. (2021).10.1016/j.trechm.2020.12.001 ↗
19. Häse, F., Roch, L. M. & Aspuru-Guzik, A. Next-generation experimentation with self-driving laboratories. Trends Chem. 1, 282–291 (2019).10.1016/j.trechm.2019.02.007 ↗
20. Ren, F. et al. Accelerated discovery of metallic glasses through iteration of machine learning and high-throughput experiments. Sci. Adv. 4, eaaq1566 (2018).10.1126/sciadv.aaq1566 ↗
21. Nikolaev, P. et al. Autonomy in materials research: a case study in carbon nanotube growth. npj Comput. Mater. 2, 1–6 (2016).10.1038/npjcompumats.2016.31 ↗
22. Herbol, H. C., Hu, W., Frazier, P., Clancy, P. & Poloczek, M. Efficient search of compositional space for hybrid organic–inorganic perovskites via bayesian optimization. npj Comput. Mater. 4, 1–7 (2018).10.1038/s41524-018-0106-7 ↗
23. Sun, S. et al. A data fusion approach to optimize compositional stability of halide perovskites. Matter 4, 1305–1322 (2021).10.1016/j.matt.2021.01.008 ↗
24. Gongora, A. E. et al. A bayesian experimental autonomous researcher for mechanical design. Sci. Adv. 6, eaaz1708 (2020).10.1126/sciadv.aaz1708 ↗
25. Häse, F., Roch, L. M., Kreisbeck, C. & Aspuru-Guzik, A. Phoenics: a bayesian optimizer for chemistry. ACS Cent. Sci. 4, 1134–1145 (2018).10.1021/acscentsci.8b00307 ↗
26. Gongora, A. E. et al. Using simulation to accelerate autonomous experimentation: a case study using mechanics. iScience 24, 102262 (2021).10.1016/j.isci.2021.102262 ↗
27. Langner, S. et al. Beyond ternary opv: high-throughput experimentation and self-driving laboratories optimize multicomponent systems. Adv. Mater. 32, 1907801 (2020).10.1002/adma.201907801 ↗
28. Rohr, B. et al. Benchmarking the acceleration of materials discovery by sequential learning. Chem. Sci. 11, 2696–2706 (2020).10.1039/C9SC05999G ↗
29. Graff, D. E., Shakhnovich, E. I. & Coley, C. W. Accelerating high-throughput virtual screening through molecular pool-based active learning. Chem. Sci. (2021).10.1039/D0SC06805E ↗
30. Hase, F. et al. Olympus: a benchmarking framework for noisy optimization and experiment planning. Mach. learn.: Sci. Technol. (2021).10.1088/2632-2153/abedc8 ↗
31. Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
32. Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).10.1023/A:1010933404324 ↗
33. Liaw, A. & Wiener, M. et al. Classification and regression by randomforest. R news 2, 18–22 (2002).
34. Neal, R. M. Bayesian learning for neural networks, vol. 118 (Springer Science & Business Media, 2012).
35. Liang, Q. Benchmarking the performance of bayesian optimization across multiple experimental materials science domains. at https://github.com/PV-Lab/Benchmarking (2021).10.1038/s41524-021-00656-9 ↗
36. GPy. GPy: A gaussian process framework in python. at http://github.com/SheffieldML/GPy (2012).
37. Reuther, A. et al. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), 1–6 (IEEE, 2018).10.1109/HPEC.2018.8547629 ↗
38. Srinivas, N., Krause, A., Kakade, S. M. & Seeger, M. W. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory 58, 3250–3265 (2012).10.1109/TIT.2011.2182033 ↗
39. Snoek, J., Larochelle, H. & Adams, R. P. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25 (2012).
40. Wu, J. & Frazier, P. Practical two-step lookahead bayesian optimization. In Advances in Neural Information Processing Systems, 32, 9813–9823 (2019).
41. Ryzhov, I. O. On the convergence rates of expected improvement methods. Oper. Res. 64, 1515–1528 (2016).10.1287/opre.2016.1494 ↗
42. Hennig, P. & Schuler, C. J. Entropy search for information-efficient global optimization. J. Mach. Learn. Res 13 (2012).
43. Frazier, P. I. Bayesian optimization (INFORMS, 2018).10.1287/educ.2018.0188 ↗
44. Roy, M.-H. & Larocque, D. Robustness of random forests for regression. J. Nonparametric Stat 24, 993–1006 (2012).10.1080/10485252.2012.715161 ↗
45. Snelson, E. & Ghahramani, Z. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems 18, 1259–1266 (2006).
46. Snelson, E. L. Flexible and efficient Gaussian process models for machine learning (University College London, 2007).
47. Snelson, E. & Ghahramani, Z. Local and global sparse gaussian process approximations. In Artificial Intelligence and Statistics, 524–531 (2007).
48. Candelieri, A., Perego, R. & Archetti, F. Bayesian optimization of pump operations in water distribution systems. J. Glob. Optim 71, 213–235 (2018).10.1007/s10898-018-0641-2 ↗
49. Wilson, A. & Adams, R. Gaussian process kernels for pattern discovery and extrapolation. In International Conference on Machine Learning, 1067–1075 (2013).
50. Schlessinger, L., Malkomes, G. & Garnett, R. Automated model search using bayesian optimization and genetic programming. In Workshop on Meta-Learning at Advances in Neural Information Processing Systems (2019).
51. Cho, Y. Kernel methods for deep learning (University of California, San Diego, 2012).
52. Paananen, T., Piironen, J., Andersen, M. R. & Vehtari, A. Variable selection for gaussian processes via sensitivity analysis of the posterior predictive distribution. In International Conference on Artificial Intelligence and Statistics, 1743–1752 (PMLR, 2019).
53. Bash, D. et al. Multi-fidelity high-throughput optimization of electrical conductivity in p3ht-cnt composites. Adv. Funct. Mater. 2102606 (2021).10.1002/adfm.202102606 ↗
54. Mekki-Berrada, F. et al. Two-step machine learning enables optimized nanoparticle synthesis. npj Comput. Mater. 7, 1–10 (2021).10.1038/s41524-021-00520-w ↗
55. Deneault, J. R. et al. Toward autonomous additive manufacturing: Bayesian optimization on a 3d printer. MRS Bull. 1–10 (2021).10.1557/s43577-021-00051-1 ↗