Benchmarking graph neural networks for materials chemistry
Source abstract
Abstract
Abstract Graph neural networks (GNNs) have received intense interest as a rapidly expanding class of machine learning models remarkably well-suited for materials applications. To date, a number of successful GNNs have been proposed and demonstrated for systems ranging from crystal stability to electronic property prediction and to surface chemistry and heterogeneous catalysis. However, a consistent benchmark of these models remains lacking, hindering the development and consistent evaluation of new models in the materials field. Here, we present a workflow and testing platform, MatDeepLearn, for quickly and reproducibly assessing and comparing GNNs and other machine learning models. We use this platform to optimize and evaluate a selection of top performing GNNs on several representative datasets in computational materials chemistry. From our investigations we note the importance of hyperparameter selection and find roughly similar performances for the top models once optimized. We identify several strengths in GNNs over conventional models in cases with compositionally diverse datasets and in its overall flexibility with respect to inputs, due to learned rather than defined representations. Meanwhile several weaknesses of GNNs are also observed including high data requirements, and suggestions for further improvement for applications in materials chemistry are discussed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Advanced Graph Neural Networks · Computational Drug Discovery Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (55)
Show source-provided references
1. Schmidt, J., Marques, M. R. G., Botti, S. & Marques, M. A. L. Recent advances and applications of machine learning in solid-state materials science. Npj Comput. Mater. 5, 83 (2019).10.1038/s41524-019-0221-0 ↗
2. Alberi, K. et al. The 2019 materials by design roadmap. J. Phys. D 52, 013001 (2018).10.1088/1361-6463/aad926 ↗
3. Chen, C., Zuo, Y., Ye, W., Li, X., Deng, Z. & Ong, S. P. A critical review of machine learning of energy materials. Adv. Energy Mater. 10, 1903242 (2020).10.1002/aenm.201903242 ↗
4. Batra, R., Song, L. & Ramprasad, R. Emerging materials intelligence ecosystems propelled by machine learning. Nat. Rev. Mater, 1–24 (2020).10.1038/s41578-020-00255-y ↗
5. Schleder, G. R., Padilha, A. C. M., Acosta, C. M., Costa, M. & Fazzio, A. From DFT to machine learning: recent approaches to materials science–a review. J. Phys. Mater. 2, 032001 (2019).10.1088/2515-7639/ab084b ↗
6. Schlexer Lamoureux, P. et al. Machine learning for computational heterogeneous catalysis. ChemCatChem 11, 3581–3601 (2019).10.1002/cctc.201900595 ↗
7. Saal, J. E., Kirklin, S., Aykol, M., Meredig, B. & Wolverton, C. Materials design and discovery with high-throughput density functional theory: the Open Quantum Materials Database (OQMD). JOM 65, 1501–1509 (2013).10.1007/s11837-013-0755-4 ↗
8. Choudhary, K. et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. Npj Comput. Mater. 6, 173 (2020).10.1038/s41524-020-00440-1 ↗
9. Curtarolo, S. et al. AFLOWLIB.ORG: a distributed materials properties repository from high-throughput ab initio calculations. Comput. Mater. Sci. 58, 227–235 (2012).10.1016/j.commatsci.2012.02.002 ↗
10. Draxl, C. & Scheffler, M. The NOMAD laboratory: from data sharing to artificial intelligence. J. Phys. Mater. 2, 036001 (2019).10.1088/2515-7639/ab13bb ↗
11. Chanussot, L. et al. The Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catal. 11, 6059–6072 (2021).10.1021/acscatal.0c04525 ↗
12. Jain, A. et al. Commentary: the materials project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).10.1063/1.4812323 ↗
13. Clement, C. L., Kauwe, S. K. & Sparks, T. D. Benchmark AFLOW data sets for machine learning. Integr. Mater. Manuf. Innov. 9, 153–156 (2020).10.1007/s40192-020-00174-4 ↗
14. Wang, A. Y.-T. et al. Machine learning for materials scientists: an introductory guide toward best practices. Chem. Mater. 32, 4954–4965 (2020).10.1021/acs.chemmater.0c01907 ↗
15. Rupp, M., Tkatchenko, A., Müller, K.-R. & von Lilienfeld, O. A. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108, 058301 (2012).10.1103/PhysRevLett.108.058301 ↗
16. Behler, J. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. 134, 074106 (2011).10.1063/1.3553717 ↗
17. De, S., Bartók, A. P., Csányi, G. & Ceriotti, M. Comparing molecules and solids across structural and alchemical space. Phys. Chem. Chem. Phys. 18, 13754–13769 (2016).10.1039/C6CP00415F ↗
18. Battaglia, P. W. et al. Relational inductive biases, deep learning, and graph networks. Preprint at https://arxiv.org/abs/1806.01261 (2018).
19. Zhou, J. et al. Graph neural networks: a review of methods and applications. AI Open 1, 57–81 (2020).10.1016/j.aiopen.2021.01.001 ↗
20. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C. & Yu, P. S. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn Syst. 32, 4–24 (2021).10.1109/TNNLS.2020.2978386 ↗
21. Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. ICML 1263-1272 (2017).
22. Schütt, K., Kindermans, P.-J., Sauceda Felix, H. E., Chmiela, S., Tkatchenko, A. & Müller, K.-R. Schnet: a continuous-filter convolutional neural network for modeling quantum interactions. NeurIPS 30, 991–1001 (2017).
23. Duvenaud, D. K. et al. Convolutional networks on graphs for learning molecular fingerprints. NeurIPS 2224–2232 (2015).
24. Wu, Z. et al. MoleculeNet: a benchmark for molecular machine learning. Chem. Sci. 9, 513–530 (2018).10.1039/C7SC02664A ↗
25. Dunn, A., Wang, Q., Ganose, A., Dopp, D. & Jain, A. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. Npj Comput. Mater. 6, 138 (2020).10.1038/s41524-020-00406-3 ↗
26. Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater. 31, 3564–3572 (2019).10.1021/acs.chemmater.9b01294 ↗
27. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).10.1103/PhysRevLett.120.145301 ↗
28. Louis, S.-Y. et al. Graph convolutional neural networks with global attention for improved materials property prediction. Phys. Chem. Chem. Phys. 22, 18141–18148 (2020).10.1039/D0CP01474E ↗
29. Park, C. W. & Wolverton, C. Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Phys. Rev. Mater. 4, 063801 (2020).10.1103/PhysRevMaterials.4.063801 ↗
30. Karamad, M., Magar, R., Shi, Y., Siahrostami, S., Gates, I. D., Barati & Farimani, A. Orbital graph convolutional neural network for material property prediction. Phys. Rev. Mater. 4, 093801 (2020).10.1103/PhysRevMaterials.4.093801 ↗
31. Back, S., Yoon, J., Tian, N., Zhong, W., Tran, K. & Ulissi, Z. W. Convolutional neural network of atomic surface structures to predict binding energies for high-throughput screening of catalysts. J. Phys. Chem. Lett. 10, 4401–4408 (2019).10.1021/acs.jpclett.9b01428 ↗
32. Palizhati, A., Zhong, W., Tran, K., Back, S. & Ulissi, Z. W. Toward predicting intermetallics surface properties with high-throughput DFT and convolutional neural networks. J. Chem. Inf. Model. 59, 4742–4749 (2019).10.1021/acs.jcim.9b00550 ↗
33. Gu, G. H., Noh, J., Kim, S., Back, S., Ulissi, Z. & Jung, Y. Practical deep-learning representation for fast heterogeneous catalyst screening. J. Phys. Chem. Lett. 11, 3185–3191 (2020).10.1021/acs.jpclett.0c00634 ↗
34. Palizhati, A., Zhong, W., Tran, K., Back, S., Ulissi, Z. W. Toward predicting intermetallics surface properties with high-throughput DFT and convolutional neural networks. J. Chem. Inf. Model. 59, 4742-4749 (2019).10.1021/acs.jcim.9b00550 ↗
35. Keith, J. A. et al. Combining machine learning and computational chemistry for predictive insights into chemical systems. Preprint at https://arxiv.org/abs/2102.06321 (2021).
36. MatDeepLearn. https://github.com/vxfung/MatDeepLearn. Accessed 1/4/2021 (2021).
37. Hestness, J. et al. Deep learning scaling is predictable, empirically. Preprint at https://arxiv.org/abs/1712.00409 (2017).
38. You, J., Ying, Z. & Leskovec, J. Design space for graph neural networks. NeurIPS 33, (2020).
39. Anthony, W, Steven, K, Ryan, M. & Taylor, S. Compositionally-restricted attention-based network for materials property prediction. Preprint at https://chemrxiv.org/articles/preprint/Compositionally-Restricted_Attention-Based_Network_for_Materials_Property_Prediction/11869026 (2020).
40. Peterson, G. G. C. & Brgoch, J. Materials discovery through machine learning formation energy. J. Phys. Energy 3, 022002 (2021).10.1088/2515-7655/abe425 ↗
41. Goodall, R. E. A. & Lee, A. A. Predicting materials properties without crystal structure: deep representation learning from stoichiometry. Nat. Commun. 11, 6280 (2020).10.1038/s41467-020-19964-7 ↗
42. Qiao, Z., Welborn, M., Anandkumar, A., Manby, F. R. & MillerIII, T. F. OrbNet: deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. J. Chem. Phys. 153, 124111 (2020).10.1063/5.0021955 ↗
43. Chen, C. et al. Learning properties of ordered and disordered materials from multi-fidelity data. Nat. Comput. Sci. 1, 46–53 (2021).10.1038/s43588-020-00002-x ↗
44. Mamun, O., Winther, K. T., Boes, J. R. & Bligaard, T. High-throughput calculations of catalytic properties of bimetallic alloy surfaces. Sci. Data 6, 76 (2019).10.1038/s41597-019-0080-z ↗
45. Rosen, A. S. et al. Machine learning the quantum-chemical properties of metal–organic frameworks for accelerated materials discovery. Matter 4, 1578–1597 (2021).10.1016/j.matt.2021.02.015 ↗
46. Haastrup, S. et al. The Computational 2D Materials Database: high-throughput modeling and discovery of atomically thin crystals. 2D Mater. 5, 042002 (2018).10.1088/2053-1583/aacfc1 ↗
47. Fung, V. & D-e, Jiang Exploring structural diversity and fluxionality of Ptn (n = 10–13) clusters from first-principles. J. Phys. Chem. C. 121, 10796–10802 (2017).10.1021/acs.jpcc.6b11968 ↗
48. Unke, O. T. et al. Machine learning force fields. Preprint at arXiv: 201007067, (2020).
49. Behler, J. Constructing high-dimensional neural network potentials: a tutorial review. Int J. Quantum Chem. 115, 1032–1050 (2015).10.1002/qua.24890 ↗
50. Kipf, T. N. & Welling, M. Semi-supervised classification with graph convolutional networks. Preprint at https://arxiv.org/abs/1609.02907 (2016).
51. Bergstra, J, Yamins, D. & Cox, D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. ICML 115–123 (2013).
52. Faber, F., Lindmaa, A., von Lilienfeld, O. A. & Armiento, R. Crystal structure representations for machine learning models of formation energies. Int. J. Quantum Chem. 115, 1094–1101 (2015).10.1002/qua.24917 ↗
53. Fey, M. & Lenssen, J. E. Fast graph representation learning with PyTorch Geometric. Preprint at arXiv:190302428 (2019).
54. Himanen, L. et al. DScribe: library of descriptors for machine learning in materials science. Comput. Phys. Commun. 247, 106949 (2020).10.1016/j.cpc.2019.106949 ↗
55. Liaw, R. et al. Tune: a research platform for distributed model selection and training. Preprint at https://arxiv.org/abs/1807.05118 (2018).