A critical examination of robustness and generalizability of machine learning prediction of materials properties
Source abstract
Abstract
Abstract Recent advances in machine learning (ML) have led to substantial performance improvement in material database benchmarks, but an excellent benchmark score may not imply good generalization performance. Here we show that ML models trained on Materials Project 2018 can have severely degraded performance on new compounds in Materials Project 2021 due to the distribution shift. We discuss how to foresee the issue with a few simple tools. Firstly, the uniform manifold approximation and projection (UMAP) can be used to investigate the relation between the training and test data within the feature space. Secondly, the disagreement between multiple ML models on the test data can illuminate out-of-distribution samples. We demonstrate that the UMAP-guided and query by committee acquisition strategies can greatly improve prediction accuracy by adding only 1% of the test data. We believe this work provides valuable insights for building databases and models that enable better robustness and generalizability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Computational Drug Discovery Methods · X-ray Diffraction in Crystallography
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (64)
Show source-provided references
1. Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O. & Walsh, A. Machine learning for molecular and materials science. Nature 559, 547–555 (2018).10.1038/s41586-018-0337-2 ↗
2. Vasudevan, R. K. et al. Materials science in the artificial intelligence age: high-throughput library generation, machine learning, and a pathway from correlations to the underpinning physics. MRS Commun. 9, 821–838 (2019).10.1557/mrc.2019.95 ↗
3. Morgan, D. & Jacobs, R. Opportunities and challenges for machine learning in materials science. Annu. Rev. Mater. Res. 50, 71–103 (2020).10.1146/annurev-matsci-070218-010015 ↗
4. DeCost, B. L. et al. Scientific AI in materials science: a path to a sustainable and scalable paradigm. Mach. Learn. Sci. Technol. 1, 033001 (2020).10.1088/2632-2153/ab9a20 ↗
5. Hart, G. L. W., Mueller, T., Toher, C. & Curtarolo, S. Machine learning for alloys. Nat. Rev. Mater. 6, 730–755 (2021).10.1038/s41578-021-00340-w ↗
6. Stach, E. et al. Autonomous experimentation systems for materials development: a community perspective. Matter 4, 2702–2726 (2021).10.1016/j.matt.2021.06.036 ↗
7. Choudhary, K. et al. Recent advances and applications of deep learning methods in materials science. npj Comput. Mater. 8, 59 (2022).10.1038/s41524-022-00734-6 ↗
8. Schleder, G. R., Padilha, A. C., Acosta, C. M., Costa, M. & Fazzio, A. From DFT to machine learning: recent approaches to materials science–a review. J. Phys. Mater. 2, 032001 (2019).10.1088/2515-7639/ab084b ↗
9. Green, M. L., Maruyama, B. & Schrier, J. Autonomous (ai-driven) materials science. Appl. Phys. Rev. 9, 030401 (2022).10.1063/5.0118872 ↗
10. Kalinin, S. V. et al. Machine learning in scanning transmission electron microscopy. Nat. Rev. Methods Primers 2, 1–28 (2022).10.1038/s43586-022-00095-w ↗
11. Krenn, M. et al. On scientific understanding with artificial intelligence. Nat. Rev. Phys 4, 761–769 (2022).10.1038/s42254-022-00518-3 ↗
12. Wilkinson, M. D. et al. The fair guiding principles for scientific data management and stewardship. Sci. Data 3, 1–9 (2016).10.1038/sdata.2016.18 ↗
13. Jain, A. et al. A high-throughput infrastructure for density functional theory calculations. Comput. Mater. Sci. 50, 2295–2310 (2011).10.1016/j.commatsci.2011.02.023 ↗
14. Saal, J. E., Kirklin, S., Aykol, M., Meredig, B. & Wolverton, C. Materials design and discovery with high-throughput density functional theory: The open quantum materials database (OQMD). JOM 65, 1501–1509 (2013).10.1007/s11837-013-0755-4 ↗
15. Garrity, K. F. & Choudhary, K. Database of wannier tight-binding hamiltonians using high-throughput density functional theory. Sci. Data 8, 1–10 (2021).10.1038/s41597-021-00885-z ↗
16. Horton, M. K., Montoya, J. H., Liu, M. & Persson, K. A. High-throughput prediction of the ground-state collinear magnetic order of inorganic materials using density functional theory. npj Comput. Mater. 5, 1–11 (2019).10.1038/s41524-019-0199-7 ↗
17. Armiento, R., Kozinsky, B., Fornari, M. & Ceder, G. Screening for high-performance piezoelectrics using high-throughput density functional theory. Phys. Rev. B 84, 014103 (2011).10.1103/PhysRevB.84.014103 ↗
18. Jain, A. et al. Commentary: the Materials Project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).10.1063/1.4812323 ↗
19. Curtarolo, S. et al. AFLOW: An automatic framework for high-throughput materials discovery. Comput. Mater. Sci. 58, 218–226 (2012).10.1016/j.commatsci.2012.02.005 ↗
20. Choudhary, K. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Comput. Mater 6, 173 (2020).10.1038/s41524-020-00440-1 ↗
21. Bartók, A. P., Kondor, R. & Csányi, G. On representing chemical environments. Phys. Rev. B 87, 184115 (2013).10.1103/PhysRevB.87.184115 ↗
22. De Jong, M. et al. A statistical learning framework for materials science: application to elastic moduli of k-nary inorganic polycrystalline compounds. Sci. Rep. 6, 1–11 (2016).10.1038/s41598-016-0001-8 ↗
23. Ouyang, R., Curtarolo, S., Ahmetcik, E., Scheffler, M. & Ghiringhelli, L. M. Sisso: a compressed-sensing method for identifying the best low-dimensional descriptor in an immensity of offered candidates. Phys. Rev. Mater. 2, 083802 (2018).10.1103/PhysRevMaterials.2.083802 ↗
24. Schütt, K. T. et al. How to represent crystal structures for machine learning: towards fast prediction of electronic properties. Phys. Rev. B 89, 1–5 (2014).10.1103/PhysRevB.89.205118 ↗
25. Faber, F., Lindmaa, A., Von Lilienfeld, O. A. & Armiento, R. Crystal structure representations for machine learning models of formation energies. Int. J. Quantum Chem. 115, 1094–1101 (2015).10.1002/qua.24917 ↗
26. Ward, L., Agrawal, A., Choudhary, A. & Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Comput. Mater. 2, 1–7 (2016).10.1038/npjcompumats.2016.28 ↗
27. Ward, L. et al. Including crystal structure attributes in machine learning models of formation energies via Voronoi tessellations. Phys. Rev. B 96, 024104 (2017).10.1103/PhysRevB.96.024104 ↗
28. Ward, L. et al. Matminer: an open source toolkit for materials data mining. Comput. Mater. Sci. 152, 60–69 (2018).10.1016/j.commatsci.2018.05.018 ↗
29. Choudhary, K., DeCost, B. & Tavazza, F. Machine learning with force-field-inspired descriptors for materials: fast screening and mapping energy landscape. Phys. Rev. Mater. 2, 083801 (2018).10.1103/PhysRevMaterials.2.083801 ↗
30. Jha, D. et al. Elemnet: deep learning the chemistry of materials from only elemental composition. Sci. Rep. 8, 1–13 (2018).10.1038/s41598-018-35934-y ↗
31. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).10.1103/PhysRevLett.120.145301 ↗
32. Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater. 31, 3564–3572 (2019).10.1021/acs.chemmater.9b01294 ↗
33. De Breuck, P. P., Hautier, G. & Rignanese, G. M. Materials property prediction for limited datasets enabled by feature selection and joint learning with MODNet. npj Comput. Mater. 7, 1–8 (2021).10.1038/s41524-021-00552-2 ↗
34. Choudhary, K. & DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Comput. Mater. 7, 185 (2021).10.1038/s41524-021-00650-1 ↗
35. Schmidt, J., Pettersson, L., Verdozzi, C., Botti, S. & Marques, M. A. Crystal graph attention networks for the prediction of stable materials. Sci. Adv. 7, eabi7948 (2021).10.1126/sciadv.abi7948 ↗
36. Ihalage, A. & Hao, Y. Formula graph self-attention network for representation-domain independent materials discovery. Adv. Sci. 9, 1–15 (2022).10.1002/advs.202200164 ↗
37. Dunn, A., Wang, Q., Ganose, A., Dopp, D. & Jain, A. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Comput. Mater. 6, 1–10 (2020).10.1038/s41524-019-0267-z ↗
38. Chen, C. & Ong, S. P. Atomsets as a hierarchical transfer learning framework for small and large materials datasets. npj Comput. Mater. 7, 1–9 (2021).10.1038/s41524-021-00639-w ↗
39. Choudhary, K. et al. Unified graph neural network force-field for the periodic table: solid state applications. Dig. Discov. 25–33 (2023). https://doi.org/10.1039/D2DD00096B.10.1039/D2DD00096B ↗
40. Chen, C. & Ong, S. P. A universal graph deep learning interatomic potential for the periodic table. Nat. Comput. Sci. 2, 718–728 (2022).10.1038/s43588-022-00349-3 ↗
41. Kong, S. et al. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. Nat. Commun. 13, 949 (2022).10.1038/s41467-022-28543-x ↗
42. Stein, H. S. Advancing data-driven chemistry by beating benchmarks. Trends Chem. 4, 682 (2022).10.1016/j.trechm.2022.05.003 ↗
43. Kirkpatrick, P. & Ellis, C. Chemical space. Nature 432, 823–823 (2004).10.1038/432823a ↗
44. Davies, D. W. et al. Computational screening of all stoichiometric inorganic materials. Chem 1, 617–627 (2016).10.1016/j.chempr.2016.09.010 ↗
45. Jia, X. et al. Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis. Nature 573, 251–255 (2019).10.1038/s41586-019-1540-5 ↗
46. Griffiths, R.-R., Schwaller, P. & Lee, A. A. Dataset bias in the natural sciences: a case study in chemical reaction prediction and synthesis design. Preprint at https://arxiv.org/abs/2105.02637 (2021).
47. De Breuck, P.-P., Evans, M. L. & Rignanese, G.-M. Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet. J. Phys. Condens. Matter 33, 404002 (2021).10.1088/1361-648X/ac1280 ↗
48. Kumagai, M. et al. Effects of data bias on machine-learning-based material discovery using experimental property data. Sci. Technol. Adv. Mater. Methods 2, 302–309 (2022).
49. Kauwe, S. K., Graser, J., Murdock, R. & Sparks, T. D. Can machine learning find extraordinary materials? Comput. Mater. Sci. 174, 109498 (2020).10.1016/j.commatsci.2019.109498 ↗
50. Xiong, Z. et al. Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation. Comput. Mater. Sci. 171, 109203 (2020).10.1016/j.commatsci.2019.109203 ↗
51. Zahrt, A. F., Henle, J. J. & Denmark, S. E. Cautionary guidelines for machine learning studies with combinatorial datasets. ACS Comb. Sci. 22, 586–591 (2020).10.1021/acscombsci.0c00118 ↗
52. Ren, F. et al. Accelerated discovery of metallic glasses through iteration of machine learning and high-throughput experiments. Sci. Adv. 4, eaaq1566 (2018).10.1126/sciadv.aaq1566 ↗
53. Meredig, B. et al. Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery. Mol. Syst. Des. Eng. 3, 819–825 (2018).10.1039/C8ME00012C ↗
54. Zhao, Z.-W., del Cueto, M. & Troisi, A. Limitations of machine learning models when predicting compounds with completely new chemistries: possible improvements applied to the discovery of new non-fullerene acceptors. Digit. Discov. 3 (2022).10.1039/D2DD00004K ↗
55. Bartel, C. J. et al. A critical examination of compound stability predictions from machine-learned formation energies. npj Comput. Mater. 6, 1–11 (2020).10.1038/s41524-020-00362-y ↗
56. George, E. P., Raabe, D. & Ritchie, R. O. High-entropy alloys. Nat. Rev. Mater. 4, 515–534 (2019).10.1038/s41578-019-0121-4 ↗
57. Chen, T. & Guestrin, C. XGBoost. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (ACM, 2016).10.1145/2939672.2939785 ↗
58. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
59. Zhang, H., Nettleton, D. & Zhu, Z. Regression-enhanced random forests. JSM Proceedings, Section on Statistical Learning and Data Science, 636–647 (American Statistical Association, 2017).
60. McInnes, L., Healy, J., Saul, N. & Großberger, L. UMAP: uniform manifold approximation and projection. J. Open Source Softw. 3, 861 (2018).10.21105/joss.00861 ↗
61. Takahashi, A., Kumagai, Y., Miyamoto, J., Mochizuki, Y. & Oba, F. Machine learning models for predicting the dielectric constants of oxides based on high-throughput first-principles calculations. Phys. Rev. Mater. 4, 103801 (2020).10.1103/PhysRevMaterials.4.103801 ↗
62. Zhang, H., Chen, W. W., Rondinelli, J. M. & Chen, W. Et-al: Entropy-targeted active learning for bias mitigation in materials data. Preprint at https://arxiv.org/abs/2211.07881 (2022).
63. Abdar, M. et al. A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243–297 (2021).10.1016/j.inffus.2021.05.008 ↗
64. Meinshausen, N. & Ridgeway, G. Quantile regression forests. J. Mach. Learn. Res 7, 983–999 (2006).