Optimal pre-train/fine-tune strategies for accurate material property predictions
Source abstract
Abstract
Abstract A pathway to overcome limited data availability in materials science is to use the framework of transfer learning, where a pre-trained (PT) machine learning model (on a larger dataset) can be fine-tuned (FT) on a target (smaller) dataset. We systematically explore the effectiveness of various PT/FT strategies to learn and predict material properties and create generalizable models by PT on multiple properties (MPT) simultaneously. Specifically, we leverage graph neural networks (GNNs) to PT/FT on seven diverse curated materials datasets, with sizes ranging from 941 to 132,752. Besides identifying optimal PT/FT strategies and hyperparameters, we find our pair-wise PT-FT models to consistently outperform models trained from scratch on target datasets. Importantly, our MPT models outperform pair-wise models on several datasets and, more significantly, on a 2D material band gap dataset that is completely out-of-domain. Finally, we expect our PT/FT and MPT frameworks to accelerate materials design and discovery for various applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · X-ray Diffraction in Crystallography · Welding Techniques and Residual Stresses
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (50)
Show source-provided references
1. Xu, P., Ji, X., Li, M. & Lu, W. Small data machine learning in materials science. npj Comput. Mater. 9, 42 (2023).10.1038/s41524-023-01000-z ↗
2. Chan, C. H., Sun, M. & Huang, B. Application of machine learning for advanced material prediction and design. EcoMat 4, e12194 (2022).10.1002/eom2.12194 ↗
3. Du, X. et al. Machine-learning-accelerated simulations to enable automatic surface reconstruction. Nat. Comput. Sci. 3, 1034–1044 (2023).10.1038/s43588-023-00571-7 ↗
4. Xian, R. P. et al. A machine learning route between band mapping and band structure. Nat. Comput. Sci. 3, 101–114 (2023).10.1038/s43588-022-00382-2 ↗
5. Witman, M. D., Goyal, A., Ogitsu, T., McDaniel, A. H. & Lany, S. Defect graph neural networks for materials discovery in high-temperature clean-energy applications. Nat. Comput. Sci. 3, 675–686 (2023).10.1038/s43588-023-00495-2 ↗
6. Dunn, A., Wang, Q., Ganose, A., Dopp, D. & Jain, A. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Comput. Mater. 6, 138 (2020).10.1038/s41524-020-00406-3 ↗
7. Omee, S. S., Fu, N., Dong, R., Hu, M. & Hu, J. Structure-based out-of-distribution (OOD) materials property prediction:a benchmark study. npj Comput. Mater. 10, 144 (2024).10.1038/s41524-024-01316-4 ↗
8. Zhao, L. & Akoglu, L. PairNorm: tackling oversmoothing in GNNS. https://arxiv.org/abs/1909.12223 (2019).
9. George, D., Shen, H. & Huerta, E. Classification and unsupervised clustering of ligo data with deep transfer learning. Phys. Rev. D. 97, 101501 (2018).10.1103/PhysRevD.97.101501 ↗
10. Kaur, T. & Gandhi, T. K. Deep convolutional neural networks with transfer learning for automated brain image classification. Mach. Vis. Appl. 31, 20 (2020).10.1007/s00138-020-01069-2 ↗
11. Liu, C., Wei, Z., Ng, D. W. K., Yuan, J. & Liang, Y.-C. Deep transfer learning for signal detection in ambient backscatter communications. IEEE Trans. Wirel. Commun. 20, 1624–1638 (2020).10.1109/TWC.2020.3034895 ↗
12. Das, N. N., Kumar, N., Kaur, M., Kumar, V. & Singh, D. Automated deep transfer learning-based approach for detection of covid-19 infection in chest x-rays. Ing. Rec. Biomed. 43, 114–119 (2022).
13. Weiss, K., Khoshgoftaar, T. M. & Wang, D. A survey of transfer learning. J. Big data 3, 1–40 (2016).10.1186/s40537-016-0043-6 ↗
14. Gupta, V. et al. Structure-aware graph neural network based deep transfer learning framework for enhanced predictive analytics on diverse materials datasets. npj Comput. Mater. 10, 1 (2024).10.1038/s41524-023-01185-3 ↗
15. Gupta, V. et al. Cross-property deep transfer learning framework for enhanced predictive analytics on small materials data. Nat. Commun. 12, 6595 (2021).10.1038/s41467-021-26921-5 ↗
16. Jha, D. et al. Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning. Nat. Commun. 10, 5316 (2019).10.1038/s41467-019-13297-w ↗
17. Jha, D. et al. ElemNet: Deep learning the chemistry of materials from only elemental composition. Sci. Rep. 8, 17593 (2018).10.1038/s41598-018-35934-y ↗
18. Kohn, W. & Sham, L. J. Self-consistent equations including exchange and correlation effects. Phys. Rev. 140, A1133 (1965).10.1103/PhysRev.140.A1133 ↗
19. Hohenberg, P. & Kohn, W. Inhomogeneous electron gas. Phys. Rev. 136, B864 (1964).10.1103/PhysRev.136.B864 ↗
20. Saal, J. E., Kirklin, S., Aykol, M., Meredig, B. & Wolverton, C. Materials design and discovery with high-throughput density functional theory: the open quantum materials database (OQMD). JOM 65, 1501–1509 (2013).10.1007/s11837-013-0755-4 ↗
21. Kirklin, S. et al. The open quantum materials database (oqmd): assessing the accuracy of DFT formation energies. npj Comput. Mater. 1, 1–15 (2015).10.1038/npjcompumats.2015.10 ↗
22. Lee, J. & Asahi, R. Transfer learning for materials informatics using crystal graph convolutional neural network. Comput. Mater. Sci. 190, 110314 (2021).10.1016/j.commatsci.2021.110314 ↗
23. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).10.1103/PhysRevLett.120.145301 ↗
24. Jain, A. et al. Commentary: The materials project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).10.1063/1.4812323 ↗
25. Choudhary, K. & DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Comput. Mater. 7, 185 (2021).10.1038/s41524-021-00650-1 ↗
26. Choudhary, K. et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Comput. Mater. 6, 173 (2020).10.1038/s41524-020-00440-1 ↗
27. Chang, R., Wang, Y.-X. & Ertekin, E. Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts framework. npj Comput. Mater. 8, 242 (2022).10.1038/s41524-022-00929-x ↗
28. Wang, Z., Dai, Z., Póczos, B. & Carbonell, J. Characterizing and avoiding negative transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11293–11302 (IEEE, 2019).10.1109/CVPR.2019.01155 ↗
29. Chen, X., Wang, S., Fu, B., Long, M. & Wang, J. Catastrophic forgetting meets negative transfer: batch spectral shrinkage for safe transfer learning. Adv. Neural Inf. Process. Syst. 32, 1908–1918 (2019).
30. Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl Acad. Sci. USA 114, 3521–3526 (2017).10.1073/pnas.1611835114 ↗
31. Ward, L. et al. Matminer: an open source toolkit for materials data mining. Comput. Mater. Sci. 152, 60–69 (2018).10.1016/j.commatsci.2018.05.018 ↗
32. Batzner, S. et al. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun. 13, 2453 (2022).10.1038/s41467-022-29939-5 ↗
33. Batatia, I., Kovacs, D. P., Simm, G., Ortner, C. & Csányi, G. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Adv. Neural Inf. Process. Syst. 35, 11423–11436 (2022).
34. Park, C. W. & Wolverton, C. Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Phys. Rev. Mater. 4, 063801 (2020).10.1103/PhysRevMaterials.4.063801 ↗
35. Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater. 31, 3564–3572 (2019).10.1021/acs.chemmater.9b01294 ↗
36. Deng, B. et al. Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nat. Mach. Intell. 5, 1031–1041 (2023).10.1038/s42256-023-00716-3 ↗
37. Schütt, K. et al. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Adv. Neural Inf. Process. Syst. 30, 992–1002 (2017).
38. Dwivedi, V. P. et al. Benchmarking graph neural networks. J. Mach. Learn. Res. 24, 1–48 (2023).
39. Petretto, G. et al. High-throughput density-functional perturbation theory phonons for inorganic materials. Sci. Data 5, 1–12 (2018).10.1038/sdata.2018.65 ↗
40. Perdew, J. P., Burke, K. & Wang, Y. Generalized gradient approximation for the exchange-correlation hole of a many-electron system. Phys. Rev. B 54, 16533 (1996).10.1103/PhysRevB.54.16533 ↗
41. De Jong, M., Chen, W., Geerlings, H., Asta, M. & Persson, K. A. A database to enable discovery and design of piezoelectric materials. Sci. Data 2, 1–13 (2015).
42. Petousis, I. et al. High-throughput screening of inorganic compounds for the discovery of novel dielectric and optical materials. Sci. Data 4, 1–12 (2017).10.1038/sdata.2016.134 ↗
43. Kingsbury, R. et al. Performance comparison of r2 scan and scan metagga density functionals for solid materials via an automated, high-throughput computational workflow. Phys. Rev. Mater. 6, 013801 (2022).10.1103/PhysRevMaterials.6.013801 ↗
44. Wang, A. Y.-T. et al. Machine learning for materials scientists: an introductory guide toward best practices. Chem. Mater. 32, 4954–4965 (2020).10.1021/acs.chemmater.0c01907 ↗
45. Kim, H. E. et al. Transfer learning for medical image classification: a literature review. BMC Med. imaging 22, 69 (2022).10.1186/s12880-022-00793-7 ↗
46. Chronopoulou, A., Baziotis, C. & Potamianos, A. An embarrassingly simple approach for transfer learning from pretrained language models. https://arxiv.org/abs/1902.10547 (2019).10.18653/v1/N19-1213 ↗
47. Sanyal, S. et al. MT-CGCNN: Integrating crystal graph convolutional neural network with multitask learning for material property prediction. https://arxiv.org/abs/1811.05660 (2018).
48. Qiao, Z. et al. Multi-task learning for electronic structure to predict and explore molecular potential energy surfaces. https://arxiv.org/abs/2011.02680 (2020).
49. Tan, Z., Li, Y., Shi, W. & Yang, S. A multitask approach to learn molecular properties. J. Chem. Inf. Model. 61, 3824–3834 (2021).10.1021/acs.jcim.1c00646 ↗
50. Ramakrishnan, R., Dral, P. O., Rupp, M. & Von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 1–7 (2014).10.1038/sdata.2014.22 ↗