DPA-2: a large atomic model as a multi-task learner
Source abstract
Abstract
The rapid advancements in artificial intelligence (AI) are catalyzing transformative changes in atomic modeling, simulation, and design. AI-driven potential energy models have demonstrated the capability to conduct large-scale, long-duration simulations with the accuracy of ab initio electronic structure methods. However, the model generation process remains a bottleneck for large-scale applications. We propose a shift towards a model-centric ecosystem, wherein a large atomic model (LAM), pre-trained across multiple disciplines, can be efficiently fine-tuned and distilled for various downstream tasks, thereby establishing a new framework for molecular modeling. In this study, we introduce the DPA-2 architecture as a prototype for LAMs. Pre-trained on a diverse array of chemical and materials systems using a multi-task approach, DPA-2 demonstrates superior generalization capabilities across multiple downstream tasks compared to the traditional single-task pre-training and fine-tuning methodologies. Our approach sets the stage for the development and broad application of LAMs in molecular and materials simulation research.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · X-ray Diffraction in Crystallography · Fuel Cells and Related Materials
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (93)
Show source-provided references
1. Hohenberg, P. & Kohn, W. Inhomogeneous electron gas. Phys. Rev. 136, B864 (1964).10.1103/PhysRev.136.B864 ↗
2. Kohn, W. & Sham, Lu. Jeu Self-consistent equations including exchange and correlation effects. Phys. Rev. 140, A1133 (1965).10.1103/PhysRev.140.A1133 ↗
3. Badaoui, M. et al. Combined free-energy calculation and machine learning methods for understanding ligand unbinding kinetics. J. Chem. Theory Comput. 18, 2543–2555 (2022).10.1021/acs.jctc.1c00924 ↗
4. Zeng, J., Tao, Y., Giese, T. J. & York, D. M. Qdπ: a quantum deep potential interaction model for drug discovery. J. Chem. Theory Comput. 19, 1261–1275 (2023).10.1021/acs.jctc.2c01172 ↗
5. Bartók, A. P., Kermode, J., Bernstein, N. & Csányi, G. ábor Machine learning a general-purpose interatomic potential for silicon. Phys. Rev. X 8, 041048 (2018).
6. Deringer, V. L., Caro, M. A. & Csányi, G. ábor A general-purpose machine-learning force field for bulk and nanostructured phosphorus. Nat. Commun. 11, 1–11 (2020).10.1038/s41467-020-19168-z ↗
7. Wen, T., Zhang, L., Wang, H., Weinan, E. & Srolovitz, D. J. Deep potentials for materials science. Mater. Futures 1, 022601 (2022).10.1088/2752-5724/ac681d ↗
8. Ma, S. & Liu, Zhi-Pan Machine learning for atomic simulation and activity prediction in heterogeneous catalysis: current status and future. ACS Catal. 10, 13213–13226 (2020).10.1021/acscatal.0c03472 ↗
9. Yang, M., Raucci, U., & Parrinello, M. Ammonia decomposition on lithium imide surfaces: a new paradigm in heterogeneous catalysis. ChemRxiv https://doi.org/10.26434/chemrxiv-2022-qr7wt (2022).10.26434/chemrxiv-2022-qr7wt ↗
10. Car, R. & Parrinello, M. Unified approach for molecular dynamics and density-functional theory. Phys. Rev. Lett. 55, 2471 (1985).10.1103/PhysRevLett.55.2471 ↗
11. Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O. & Roitberg, A. E. Less is more: sampling chemical space with active learning. J. Chem. Phys. 148, 241733 (2018).10.1063/1.5023802 ↗
12. Uteva, E., Graham, R. S., Wilkinson, R. D. & Wheatley, R. J. Active learning in gaussian process interpolation of potential energy surfaces. J. Chem. Phys. 149, 174114 (2018).10.1063/1.5051772 ↗
13. Zhang, L., Lin, De-Ye, Wang, H., Car, R. & Weinan, E. Active learning of uniformly accurate interatomic potentials for materials simulation. Phys. Rev. Mater. 3, 023804 (2019).10.1103/PhysRevMaterials.3.023804 ↗
14. Zhang, Y. et al. Dp-gen: a concurrent learning platform for the generation of reliable deep learning based potential energy models. Comput. Phys. Commun. 253, 107206 (2020).10.1016/j.cpc.2020.107206 ↗
15. Jiang, W., Zhang, Y., Zhang, L. & Wang, H. Accurate deep potential model for the Al–Cu–Mg alloy in the full concentration space. Chin. Phys. B 30, 050706 (2021).10.1088/1674-1056/abf134 ↗
16. Takamoto, S. et al. Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements. Nat. Commun. 13, 2991 (2022).10.1038/s41467-022-30687-9 ↗
17. Chen, C. & Ong, Shyue Ping. A universal graph deep learning interatomic potential for the periodic table. Nat. Comput. Sci. 2, 718–728 (2022).10.1038/s43588-022-00349-3 ↗
18. Choudhary, K. et al. Unified graph neural network force-field for the periodic table: solid state applications. Digit. Discov. 2, 346–355 (2023).10.1039/D2DD00096B ↗
19. Deng, B. et al. Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nat. Mach. Intell. 5, 1031–1041 (2023).10.1038/s42256-023-00716-3 ↗
20. Batatia, I. et al. A foundation model for atomistic materials chemistry. Preprint at arXiv https://doi.org/10.48550/arXiv.2401.00096 (2023).10.48550/arXiv.2401.00096 ↗
21. Merchant, A. et al. Scaling deep learning for materials discovery. Nature 624, 1–6 (2023).10.1038/s41586-023-06735-9 ↗
22. Jain, A. et al. The materials project: a materials genome approach to accelerating materials innovation. APL Mater 1, 011002 (2013).10.1063/1.4812323 ↗
23. Perdew, J. P., Burke, K. & Ernzerhof, M. Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865 (1996).10.1103/PhysRevLett.77.3865 ↗
24. Chai, J.-D. & Head-Gordon, M. Systematic optimization of long-range corrected hybrid density functionals. J. Chem. Phys. 128, 084106 (2008).10.1063/1.2834918 ↗
25. Cui, T. et al. Gpip: geometry-enhanced pre-training on interatomic potentials. Preprint at arXiv https://doi.org/10.48550/arXiv.2309.15718 (2023).10.48550/arXiv.2309.15718 ↗
26. Feng, R. et al. May the force be with you: unified force-centric pre-training for 3d molecular conformations. Adv. Neural Inf. Process. Syst. 36, (2024).
27. Jacobson, L., Stevenson, J., Ramezanghorbani, F., Dajnowicz, S., & Leswing, K. Leveraging multitask learning to improve the transferability of machine learned force fields. ChemRxiv, (2023).10.26434/chemrxiv-2023-8n737 ↗
28. Wang, Y., Xu, C., Li, Z. & Farimani, A. B. Denoise pre-training on non-equilibriummolecules for accurate and transferable neuralpotentials. J. Chem. Theory Comput. 19, 5077–5087 (2023).10.1021/acs.jctc.3c00289 ↗
29. Kokkinos, I. Ubernet: training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of Proceedings of the IEEE conference on computer vision and pattern recognition 6129–6138 (CVPR, 2017).10.1109/CVPR.2017.579 ↗
30. Devlin, J., Chang, Ming-Wei, Lee, K., & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. Preprint at arXiv https://doi.org/10.48550/arXiv.1810.04805 (2018).10.48550/arXiv.1810.04805 ↗
31. Shoghi, N. et al. From molecules to materials: pre-training large generalizable models for atomic property prediction. In Proceedings of The Twelfth International Conference on Learning Representations, (2024).
32. Smith, J. S. et al. Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning. Nat. Commun. 10, 1–8 (2019).10.1038/s41467-019-10827-4 ↗
33. Kolluru, A. et al. Transfer learning using attentions across atomic systems with graph neural networks (taag). J. Chem. Phys. 156, 184702 (2022).10.1063/5.0088019 ↗
34. Goodfellow, I., Bengio, Y., & Courville, A. Deep Learning (MIT Press, 2016).
35. Tuckerman, M. Statistical Mechanics: Theory and Molecular Simulation (OUP Oxford, 2010).
36. Behler, J. örg & Parrinello, M. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett. 98, 146401 (2007).10.1103/PhysRevLett.98.146401 ↗
37. Smith, J. S., Isayev, O. & Roitberg, A. E. Ani-1: an extensible neural network potential with dft accuracy at force field computational cost. Chem. Sci. 8, 3192–3203 (2017).10.1039/C6SC05720A ↗
38. Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R. & Tkatchenko, A. Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 1–8 (2017).10.1038/ncomms13890 ↗
39. Gastegger, M., Schwiedrzik, L., Bittermann, M., Berzsenyi, F., & Marquetand, P. wACSF–weighted atom-centered symmetry functions as descriptors in machine learning potentials. J. Chem. Phys. 148, 241709 (2018).10.1063/1.5019667 ↗
40. Zhang, L., Han, J., Wang, H., Car, R. & Weinan, E. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Phys. Rev. Lett. 120, 143001 (2018).10.1103/PhysRevLett.120.143001 ↗
41. Zhang, L. et al. End-to-end symmetry preserving inter-atomic potential energy model for finite and extended systems. Adv. Neural Inf. Process. Syst. 31, 4441–4451 (2018).
42. Zeng, J. et al. Deepmd-kit v2: a software package for deep potential models. J. Chem. Phys. 159, 054801 (2023).10.1063/5.0155600 ↗
43. Zhang, D. et al. Pretraining of attention-based deep learning potential model for molecular simulation. npj Comput. Mater. 10, 94 (2024).10.1038/s41524-024-01278-7 ↗
44. Zhang, Y., Hu, C. & Jiang, B. Embedded atom neural network potentials: Efficient and accurate machine learning with a physically inspired representation. J. Phys. Chem. Lett. 10, 4962–4967 (2019).10.1021/acs.jpclett.9b02037 ↗
45. Bartók, A. P., Payne, M. C., Kondor, R. & Csányi, G. ábor Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett. 104, 136403 (2010).10.1103/PhysRevLett.104.136403 ↗
46. Bartók, A. P., Kondor, R. & Csányi, G. ábor On representing chemical environments. Phys. Rev. B 87, 184115 (2013).10.1103/PhysRevB.87.184115 ↗
47. Rupp, M., Tkatchenko, A., Müller, Klaus-Robert & Von Lilienfeld, O. A. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108, 058301 (2012).10.1103/PhysRevLett.108.058301 ↗
48. Chmiela, S. et al. Machine learning of accurate energy-conserving molecular force fields. Sci. Adv. 3, e1603015 (2017).10.1126/sciadv.1603015 ↗
49. Thompson, A. P., Swiler, L. P., Trott, C. R., Foiles, S. M. & Tucker, G. J. Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials. J. Comput. Phys. 285, 316–330 (2015).10.1016/j.jcp.2014.12.018 ↗
50. Shapeev, A. V. Moment tensor potentials: a class of systematically improvable interatomic potentials. Multiscale Model. Simul. 14, 1153–1173 (2016).10.1137/15M1054183 ↗
51. Thomas, N. et al. Tensor field networks: rotation-and translation-equivariant neural networks for 3d point clouds. Preprint at arXiv https://doi.org/10.48550/arXiv.1802.08219 (2018).10.48550/arXiv.1802.08219 ↗
52. Batzner, S. et al. Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun. 13, 1–11 (2022).10.1038/s41467-022-29939-5 ↗
53. Schütt, K. et al. Schnet: a continuous-filter convolutional neural network for modeling quantum interactions. Adv. Neural Inf. Process. Syst. 30, 992–1002 (2017).
54. Gasteiger, J., Giri, S., Margraf, J. T. & Günnemann, S. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. Preprint at arXiv https://doi.org/10.48550/arXiv.2011.14115 (2020).10.48550/arXiv.2011.14115 ↗
55. Schütt, K. T., Unke, O. T. & Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In Proceedings of International Conference on Machine Learning, 9377–9388 (PMLR, 2021).
56. Gasteiger, J., Becker, F. & Günnemann, S. Gemnet: Universal directional graph neural networks for molecules. Adv. Neural Inf. Process. Syst. 34, 6790–6802 (2021).
57. Shuaibi, M. et al. Rotation invariant graph neural networks using spin convolutions. Preprint at arXiv https://doi.org/10.48550/arXiv.2106.09575 (2021).10.48550/arXiv.2106.09575 ↗
58. Zitnick, L. et al. Spherical channels for modeling atomic interactions. Adv. Neural Inf. Process. Syst. 35, 8054–8067 (2022).
59. Batzner, S. et al. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun. 13, 1–11 (2022).10.1038/s41467-022-29939-5 ↗
60. Batatia, I., Kovacs, D. P., Simm, G., Ortner, C. & Csányi, G. ábor Mace: higher order equivariant message passing neural networks for fast and accurate force fields. Adv. Neural Inf. Process. Syst. 35, 11423–11436 (2022).
61. Liao, Y.-L. & Smidt, T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. Preprint at arXiv https://doi.org/10.48550/arXiv.2206.11990 (2022).10.48550/arXiv.2206.11990 ↗
62. Liao, Y.-L., Wood, B., Das, A., & Smidt, T. Equiformerv2: improved equivariant transformer for scaling to higher-degree representations. Preprint at arXiv https://doi.org/10.48550/arXiv.2306.12059 (2023).10.48550/arXiv.2306.12059 ↗
63. Musaelian, A. et al. Learning local equivariant representations for large-scale atomistic dynamics. Nat. Commun. 14, 579 (2023).10.1038/s41467-023-36329-y ↗
64. Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).10.1109/TPAMI.2013.50 ↗
65. Zhang, D., Yin, J., Zhu, X. & Zhang, C. Network representation learning: a survey. IEEE Trans. Big Data 6, 3–28 (2018).10.1109/TBDATA.2018.2850013 ↗
66. Radford, A. et al. Improving Language Understanding by Generative Pre-training (OpenAI Blog, 2018).
67. Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations, (2021).
68. Zhou, G. et al. Uni-mol: a universal 3D molecular representation learning framework. ChemRxiv https://doi.org/10.26434/chemrxiv-2022-jjm0j-v4 (2022).10.26434/chemrxiv-2022-jjm0j-v4 ↗
69. Zhu, J. et al. Unified 2D and 3D pre-training of molecular representations. In Proceedings of Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2626–2636 (ACM, 2022).10.1145/3534678.3539368 ↗
70. Zaidi, S. et al. Pre-training via denoisingfor molecular property prediction. In Proceedings of NeurIPS 2022 AI for Science: Progress and Promises, (2022).
71. Lu, S., Gao, Z., He, D., Zhang, L., & Ke, G. Highly accurate quantum chemical propertyprediction with uni-mol+. Preprint at arXiv https://doi.org/10.48550/arXiv.2303.16982 (2023).10.48550/arXiv.2303.16982 ↗
72. Feng, S., Ni, Y., Lan, Y., Ma, Z. & Ma, W.-Y. Fractional denoising for 3D molecular pre-training. In Proceedings of International Conference on Machine Learning, 9938–9961 (PMLR, 2023).
73. Jiao, R., Han, J., Huang, W., Rong, Y. & Liu, Y. Energy-motivated equivariant pretraining for 3d molecular graphs. In Proceedings of Proceedings of the AAAI Conference on Artificial Intelligence, 378096–8104 (2023).10.1609/aaai.v37i7.25978 ↗
74. Beaini, D. et.al. Towards foundational models for molecular learning on large-scale multi-task datasets. In Proceedings of The Twelfth International Conference on Learning Representations, (2024).
75. Lee, K. L. K. et al. Towards foundation models for materials science: The open matsci ml toolkit. In Proceedings of Proceedings of the SC’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, 51–59 (2023).10.1145/3624062.3626081 ↗
76. Gardner, John LA, Baker, K. T., & Deringer, V. L. Synthetic pre-training for neural-network interatomic potentials. Mach. Learn. Sci. Technol. https://doi.org/10.1088/2632-2153/ad1626 (2023).10.1088/2632-2153/ad1626 ↗
77. Chanussot, L. et al. Open catalyst 2020 (OC20) dataset and community challenges. ACS Catal. 11, 6059–6072 (2021).10.1021/acscatal.0c04525 ↗
78. Lu, D. et al. Dp compress: a model compression scheme for generating efficient deep potential models. J. Chem. Theory Comput. 18, 5559–5567 (2022).10.1021/acs.jctc.2c00102 ↗
79. Kresse, G. & Furthmüller, J. ürgen Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Comput. Mater. Sci. 6, 15–50 (1996).10.1016/0927-0256(96)00008-0 ↗
80. Kresse, G. & Furthmüller, J. ürgen Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Phys. Rev. B 54, 11169 (1996).10.1103/PhysRevB.54.11169 ↗
81. Frisch, M.J. et al. Gaussian 16, Revision B.01 (Gaussian, Inc., 2016).
82. Chen, M., Guo, G. C. & He, L. Systematically improvable optimized atomic basis sets for ab initio calculations. J. Phys. Condens. Matter 22, 445501 (2010).10.1088/0953-8984/22/44/445501 ↗
83. Li, P. et al. Large-scale ab initio simulations based on systematically improvable atomic basis. Comput. Mater. Sci. 112, 503–517 (2016).10.1016/j.commatsci.2015.07.004 ↗
84. Gasteiger, J. et al. Gemnet-OC: Developing graph neural networks for large and diverse molecular simulation datasets. Transactions on Machine Learning Research, (2022).
85. Wu, J. et al. Universal interatomic potential for perovskite oxides. Phys. Rev. B 108, L180104 (2023).10.1103/PhysRevB.108.L180104 ↗
86. Mo, Y., Ong, ShyuePing & Ceder, G. First principles study of the li10gep2s12 lithium super ionic conductor material. Chem. Mater. 24, 15–17 (2012).10.1021/cm203303y ↗
87. Marcolongo, A. & Marzari, N. Ionic correlations and failure of nernst-einstein relation in solid-state electrolytes. Phys. Rev. Mater. 1, 025402 (2017).10.1103/PhysRevMaterials.1.025402 ↗
88. Kuhn, A., Köhler, J. ürgen & Lotsch, B. V. Single-crystal X-ray structure analysis of the superionic conductor li 10 gep 2 s 12. Phys. Chem. Chem. Phys. 15, 11620–11622 (2013).10.1039/c3cp51985f ↗
89. Huang, J. et al. Deep potential generation scheme and simulation protocol for the li10gep2s12-type superionic conductors. J. Chem. Phys. 154, 094703 (2021).10.1063/5.0041849 ↗
90. Wang, D., Cao, M. & Zhang, S. Phase diagram and properties of Pb (In1/2Nb1/2) O3–Pb (Mg1/3Nb2/3) O3–PbTiO3 polycrystalline ceramics. J. Eur. Ceram. Soc. 32, 433–439 (2012).10.1016/j.jeurceramsoc.2011.08.025 ↗
91. Li, Q. et al. Soft phonon modes and diffuse scattering in Pb (In1/2Nb1/2) O3–Pb (Mg1/3Nb2/3) O3–PbTiO3 relaxor ferroelectrics. J. Mater. 4, 345–352 (2018).
92. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https://doi.org/10.48550/arXiv.1412.6980 (2014).10.48550/arXiv.1412.6980 ↗
93. DiStasio, R. A., Santra, B., Li, Z., Wu, X., & Car, R. The individual and collective effects of exact exchange and dispersion interactions on the ab initio structure of liquid water. J. Chem. Phys. 141, 084502 (2014).10.1063/1.4893377 ↗