JARVIS-Leaderboard: a large scale benchmark of materials design methods
Source abstract
Abstract
Abstract Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard/
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Electronic and Structural Properties of Oxides · X-ray Diffraction in Crystallography
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Ward, C. H. & Warren, J. A. Materials genome initiative: materials data (US Department of Commerce, National Institute of Standards and Technology, 2015).
2. Callister, W. D. et al. Fundamentals of materials science and engineering, Vol. 471660817 (Wiley London, 2000).
3. Chen, L.-Q. Phase-field models for microstructure evolution. Annu. Rev. Mat. Res. 32, 113–140 (2002).10.1146/annurev.matsci.32.112001.132041 ↗
4. Agrawal, A., Gopalakrishnan, K., & Choudhary, A. Materials image informatics using deep learning, in Handbook on Big Data and Machine Learning in the Physical Sciences: Volume 1. Big Data Methods in Experimental Materials Discovery, series and number World Scientific Series on Emerging Technologies, edited by (WorldScientific, 2020) pp. 205–230.10.1142/9789811204555_0006 ↗
5. Choudhary, K. et al. Recent advances and applications of deep learning methods in materials science. npj Comp. Mat. 8, 59 (2022).10.1038/s41524-022-00734-6 ↗
6. Audus, D. J. et al. Artificial intelligence for materials, in https://doi.org/10.1142/9789811265679_0023Artificial Intelligence for Science, Chapter 23, pp. 413–430.10.1142/9789811265679_0023 ↗
7. Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018).10.1038/s41562-018-0399-z ↗
8. Fanelli, D. Is science really facing a reproducibility crisis, and do we need it to? Proc. Nat. Acad. Sci. 115, 2628–2631 (2018).10.1073/pnas.1708272114 ↗
9. Sun, Z. et al. Are we evaluating rigorously? Benchmarking recommendation for reproducible evaluation and fair comparison. Proc. 14th ACM Conf. on Recomm. Sys. (2020).10.1145/3383313.3412489 ↗
10. Amrhein, V., Korner-Nievergelt, Fränzi & Roth, T. The earth is flat (p> 0.05): significance thresholds and the crisis of unreplicable research. PeerJ 5, e3544 (2017).10.7717/peerj.3544 ↗
11. Grimes, DavidRobert, Bauch, C. T. & Ioannidis, JohnP. A. Modelling science trustworthiness under publish or perish pressure. Roy. Soc. Open Sci. 5.1, 171511 (2018).10.1098/rsos.171511 ↗
12. Allen, G. I., Gan, L., & Zheng, L. Interpretable Machine Learning for Discovery: Statistical Challenges and Opportunities. Ann. Rev. Stat. and App. 11 (2023).10.1146/annurev-statistics-040120-030919 ↗
13. Prager, E. M. et al. Improving transparency and scientific rigor in academic publishing. J. Neuro. Res. 97, 377–390 (2019).10.1002/jnr.24340 ↗
14. Papadiamantis, A. G. et al. Metadata stewardship in nanosafety research: community-driven organisation of metadata schemas to support FAIR nanoscience data. Nanomat 10, 2033 (2020).10.3390/nano10102033 ↗
15. Hao-Nan, Z. & Rubio-González, C. On the reproducibility of software defect datasets, 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) IEEE (2023).
16. Lehtola, S. & Marques, M. Reproducibility of density functional approximations: How new functionals should be reported. J. Chem. Phys. 159, 114116 (2023).10.1063/5.0167763 ↗
17. Sayre, F. & Riegelman, A. The reproducibility crisis and academic libraries. Coll. Res. Lib. 79, 2 (2018).
18. Papadiamantis, A. G., Ward, L. & Hattrick-Simpers, J. Metadata stewardship in nanosafety research: Community-driven organisation of metadata schemas to support FAIR nanoscience data. Dig. Disc. 3, 281–286 (2024).
19. Park, J., Howe, J. D. & Sholl, D. S. How reproducible are isotherm measurements in metal–organic frameworks? Chem. Mat. 29, 10487–10495 (2017).10.1021/acs.chemmater.7b04287 ↗
20. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).10.1038/533452a ↗
21. Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).10.1126/science.359.6377.725 ↗
22. Wilkinson, M. D. et al. The fair guiding principles for scientific data management and stewardship. Sci. Data 3, 1–9 (2016).10.1038/sdata.2016.18 ↗
23. Agrawal, A. & Choudhary, A. Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science. APL Mat. 4, 053208 (2016).10.1063/1.4946894 ↗
24. Rickman, J., Lookman, T. & Kalinin, S. Materials informatics: From the atomic-level to the continuum. Acta Mat. 168, 473–510 (2019).10.1016/j.actamat.2019.01.051 ↗
25. Agrawal, A. & Choudhary, A. Deep materials informatics: Applications of deep learning in materials science. MRS Comm. 9, 779–792 (2019).10.1557/mrc.2019.73 ↗
26. Gupta, V., Liao, W.-k, Choudhary, A. & Agrawal, A. Evolution of artificial intelligence for application in contemporary materials science. MRS Comm. 13, 754–763 (2023).10.1557/s43579-023-00433-3 ↗
27. Lejaeghere, K. et al. Reproducibility in density functional theory calculations of solids. Science 351, aad3000 (2016).10.1126/science.aad3000 ↗
28. Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comp. Vis. 115, 211–252 (2015).10.1007/s11263-015-0816-y ↗
29. Jumper, J. et al. Highly accurate protein structure prediction with alphafold. Nature 596, 583–589 (2021).10.1038/s41586-021-03819-2 ↗
30. Brown, T. et al. Language models are few-shot learners. Adv. Neur. Info Proc. Sys. 33, 1877–1901 (2020).
31. Zhang, X. et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. Preprint at https://arxiv.org/abs/2307.08423 (2023).
32. Bosoni, E. et al. How to verify the precision of density-functional-theory implementations via reproducible and universal workflows. Nat. Rev. Phys. 6, 45–58 (2024).10.1038/s42254-023-00655-3 ↗
33. Dunn, A., Wang, Q., Ganose, A., Dopp, D. & Jain, A. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Comp. Mat. 6, 138 (2020).10.1038/s41524-020-00406-3 ↗
34. Wu, Z. et al. Moleculenet: a benchmark for molecular machine learning. Chem. Sci. 9, 513–530 (2018).10.1039/C7SC02664A ↗
35. Chanussot, L. et al. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catal. 11, 6059–6072 (2021).10.1021/acscatal.0c04525 ↗
36. Chmiela, S. et al. Machine learning of accurate energy-conserving molecular force fields. Sci. Adv. 3, e1603015 (2017).10.1126/sciadv.1603015 ↗
37. Chmiela, S., Sauceda, H. E., Poltavsky, I., Müller, K.-R. & Tkatchenko, A. sgdml: Constructing accurate and data efficient molecular force fields using machine learning. Comp. Phys. Comm. 240, 38–45 (2019).10.1016/j.cpc.2019.02.007 ↗
38. Zuo, Y. et al. Performance and cost assessment of machine learning interatomic potentials. J. Phys. Chem. A 124, 731–745 (2020).10.1021/acs.jpca.9b08723 ↗
39. Weston, L. et al. Named entity recognition and normalization applied to large-scale information extraction from the materials science literature. J. Chem. Inf. Model. 59, 3692–3702 (2019).10.1021/acs.jcim.9b00470 ↗
40. Ziatdinov, M., Ghosh, A., ChunYin(Tommy), W. & Kalinin, S. V. AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy. Nat. Mach. Intel. 4, 1101–1112 (2022).10.1038/s42256-022-00555-8 ↗
41. Borlido, P. et al. Large-scale benchmark of exchange–correlation functionals for the determination of electronic band gaps of solids. J. Chem. Theor. Comp. 15, 5069–5079 (2019).10.1021/acs.jctc.9b00322 ↗
42. Huber, S. P. et al. Common workflows for computing material properties using different quantum engines. npj Comp. Mat. 7, 136 (2021).10.1038/s41524-021-00594-6 ↗
43. Zhang, G.-X., Reilly, A. M., Tkatchenko, A. & Scheffler, M. Performance of various density-functional approximations for cohesive properties of 64 bulk solids. N. J. Phys. 20, 063020 (2018).10.1088/1367-2630/aac7f0 ↗
44. Tran, R. et al. The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts. ACS Catal. 13, 3066–3084 (2023).10.1021/acscatal.2c05426 ↗
45. Jurečka, P., Šponer, J., Černy`, J. & Hobza, P. Benchmark database of accurate (mp2 and ccsd (t) complete basis set limit) interaction energies of small model complexes, dna base pairs, and amino acid pairs. Phy. Chem. Chem. Phys. 8, 1985–1993 (2006).10.1039/B600027D ↗
46. Brauer, B., Kesharwani, M. K., Kozuch, S. & Martin, J. M. The s66 × 8 benchmark for noncovalent interactions revisited: Explicitly correlated ab initio methods and density functional theory. Phys. Chem. Chem. Phys. 18, 20905–20925 (2016).10.1039/C6CP00688D ↗
47. Mata, R. A. & Suhm, M. A. Benchmarking quantum chemical methods: Are we heading in the right direction? Angew. Chem. Int. Ed. 56, 11011–11018 (2017).10.1002/anie.201611308 ↗
48. Taylor, D. E. et al. Blind test of density-functional-based methods on intermolecular interaction energies. J. Chem. Phys. 145, 124105 (2016).10.1063/1.4961095 ↗
49. Wheeler, D. et al. Pfhub: the phase-field community hub. J. Open Res. Soft. 7, 29 (2019).10.5334/jors.276 ↗
50. Lindsay, A. D. et al. 2.0 - MOOSE: Enabling massively parallel multiphysics simulation. SoftwareX 20, 101202 (2022).10.1016/j.softx.2022.101202 ↗
51. Wei, J. et al. Benchmark Tests of Atom Segmentation Deep Learning Models with a Consistent Dataset. Micro Microanal. 29, 552–562 (2023).10.1093/micmic/ozac043 ↗
52. Ren, J. et al. Diligent102: A photometric stereo benchmark dataset with controlled shape and material variation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) pp. 12581–12590.10.1109/CVPR52688.2022.01225 ↗
53. Li, M. et al. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Trans. Im. Proc. 29, 4159–4173 (2020).10.1109/TIP.2020.2968818 ↗
54. Henderson, A. N., Kauwe, S. K. & Sparks, T. D. Benchmark datasets incorporating diverse tasks, sample sizes, material systems, and data heterogeneity for materials informatics. Data Brief. 37, 107262 (2021).10.1016/j.dib.2021.107262 ↗
55. Fung, V., Zhang, J., Juarez, E. & Sumpter, B. G. Benchmarking graph neural networks for materials chemistry. npj Comp. Mat. 7, 84 (2021).10.1038/s41524-021-00554-0 ↗
56. Cecen, A., Dai, H., Yabansu, Y. C., Kalidindi, S. R. & Song, L. Material structure-property linkages using three-dimensional convolutional neural networks. Acta Mat. 146, 76–84 (2018).10.1016/j.actamat.2017.11.053 ↗
57. Baird, S. G., Issa, R. & Sparks, T. D. Materials science optimization benchmark dataset for multi-objective, multi-fidelity optimization of hard-sphere packing simulations. Data Brief. 50, 109487 (2023).10.1016/j.dib.2023.109487 ↗
58. Chen, L., Tran, H., Batra, R., Kim, C. & Ramprasad, R. Machine learning models for the lattice thermal conductivity prediction of inorganic materials. Comp. Mat. Sci. 170, 109155 (2019).10.1016/j.commatsci.2019.109155 ↗
59. Tian, S. et al. Quartet protein reference materials and datasets for multi-platform assessment of label-free proteomics. Genome Bio. 24, 202 (2023).10.1186/s13059-023-03048-y ↗
60. Fu, N. et al. Materials transformers language models for generative materials design: a benchmark study. Preprint at https://arxiv.org/abs/2206.13578 (2022).
61. Meredig, B. et al. Can machine learning identify the next high-temperature superconductor? examining extrapolation performance for materials discovery. Mol. Syst. Des. Eng. 3, 819–825 (2018).10.1039/C8ME00012C ↗
62. Lejeune, E. Mechanical mnist: A benchmark dataset for mechanical metamodels. Ext. Mech. Lett. 36, 100659 (2020).10.1016/j.eml.2020.100659 ↗
63. Clement, C. L., Kauwe, S. K. & Sparks, T. D. Benchmark aflow data sets for machine learning. Int. Mat. Manufact. Innov. 9, 153–156 (2020).10.1007/s40192-020-00174-4 ↗
64. Varivoda, D., Dong, R., Omee, S. S. & Hu, J. Materials property prediction with uncertainty quantification: A benchmark study. Appl. Phys. Rev. 10, 021409 (2023).10.1063/5.0133528 ↗
65. Jain, A. et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL Mat. 1, 011002 (2013).10.1063/1.4812323 ↗
66. Li, K., DeCost, B., Choudhary, K., Greenwood, M. & Hattrick-Simpers, J. A critical examination of robustness and generalizability of machine learning prediction of materials properties. npj Comp. Mat. 9, 55 (2023).10.1038/s41524-023-01012-9 ↗
67. Li, K. et al. Exploiting redundancy in large materials datasets for efficient machine learning with less data. Nat. Commun. 14, 7283 (2023).10.1038/s41467-023-42992-y ↗
68. Choudhary, K. & Sumpter, B. G. Can a deep-learning model make fast predictions of vacancy formation in diverse materials? AIP Adv. 13 (2023).10.1063/5.0135382 ↗
69. Vuorio, R., Sun, S.-H., Hu, H. & Lim, J. J. Multimodal model-agnostic meta-learning via task-aware modulation, in Advances in Neural Information Processing Systems, Vol. 32, (eds H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, & R. Garnett) (CurranAssociates, Inc., 2019). https://proceedings.neurips.cc/paper_files/paper/2019/file/e4da3b7fbbce2345d7772b0674a318d5-Paper.pdf.
70. Choudhary, K. et al. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj Comp. Mat. 6, 173 (2020).10.1038/s41524-020-00440-1 ↗
71. Wines, D. et al. Recent progress in the JARVIS infrastructure for next-generation data-driven materials design. Appl. Phys. Rev. 10, 041302 (2023).10.1063/5.0159299 ↗
72. Garrity, K. F. & Choudhary, K. Fast and accurate prediction of material properties with three-body tight-binding model for the periodic table. Phys. Rev. Mat. 7, 044603 (2023).
73. Reiser, P., Eberhard, A. & Friederich, P. Graph neural networks in tensorflow-keras with raggedtensor representation (kgcnn). Soft. Imp. 9, 100095 (2021).10.1016/j.simpa.2021.100095 ↗
74. Lin, Y. et al. Efficient approximations of complete interatomic potentials for crystal property prediction, in Proceedings of the 40th International Conference on Machine Learning (2023).
75. Yan, K., Liu, Y., Lin, Y., & Ji, S. Periodic graph transformers for crystal material property prediction, in The 36th Annual Conference on Neural Information Processing Systems (2022) pp. 15066–15080.
76. Choudhary, K. & DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Comp. Mat. 7, 185 (2021).10.1038/s41524-021-00650-1 ↗
77. Gupta, V. et al. Structure-aware graph neural network based deep transfer learning framework for enhanced predictive analytics on diverse materials datasets. npj Comp. Mat. 10, 1 (2024).10.1038/s41524-023-01185-3 ↗
78. Kuisma, M., Ojanen, J., Enkovaara, J. & Rantala, T. T. Kohn-sham potential with discontinuity for band gap materials. Phys. Rev. B 82, 115106 (2010).10.1103/PhysRevB.82.115106 ↗
79. Enkovaara, J. et al. Electronic structure calculations with gpaw: a real-space implementation of the projector augmented-wave method. J. Phys.: Cond. Matt. 22, 253202 (2010).
80. Onida, G., Reining, L. & Rubio, A. Electronic excitations: density-functional versus many-body green’s-function approaches. Rev. Mod. Phys. 74, 601–659 (2002).10.1103/RevModPhys.74.601 ↗
81. Kresse, G. & Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Phys. Rev. B 54, 11169 (1996).10.1103/PhysRevB.54.11169 ↗
82. Kresse, G. & Furthmüller, J. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Comp. Mat. Sci. 6, 15–50 (1996).10.1016/0927-0256(96)00008-0 ↗
83. Tran, F. & Blaha, P. Importance of the kinetic energy density for band gap calculations in solids with density functional theory. J. Phys. Chem. A 121, 3318–3325 (2017).10.1021/acs.jpca.7b02882 ↗
84. Rai, D. P., Ghimire, M. P. & Thapa, R. K. A dft study of bex (x = s, se, te) semiconductor: Modified becke johnson (mbj) potential. Semicond 48, 1411–1422 (2014).10.1134/S1063782614110244 ↗
85. Foulkes, W. M. C., Mitas, L., Needs, R. J. & Rajagopal, G. Quantum Monte Carlo simulations of solids. Rev. Mod. Phys. 73, 33–83 (2001).10.1103/RevModPhys.73.33 ↗
86. Kim, J. et al. Qmcpack: an open source ab initio quantum monte carlo package for the electronic structure of atoms, molecules and solids. J. Phys.: Cond. Matt. 30, 195901 (2018).
87. Perdew, J. P., Burke, K. & Ernzerhof, M. Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865–3868 (1996).10.1103/PhysRevLett.77.3865 ↗
88. Saal, J. E. et al. Materials design and discovery with high-throughput density functional theory: The open quantum materials database (oqmd). JOM 65, 1501–1509 (2013).10.1007/s11837-013-0755-4 ↗
89. Kirklin, S. et al. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. npj Comp. Mat. 1, 15010 (2015).10.1038/npjcompumats.2015.10 ↗
90. Curtarolo, S. et al. Aflow: An automatic framework for high-throughput materials discovery. Comp. Mat. Sci. 58, 218–226 (2012).10.1016/j.commatsci.2012.02.005 ↗
91. Choudhary, K. et al. Unified graph neural network force-field for the periodic table: solid state applications. Dig. Disc. 2, 346–355 (2023).
92. Chen, C. & Ong, S. P. A universal graph deep learning interatomic potential for the periodic table. Nat. Comp. Sci. 2, 718–728 (2022).10.1038/s43588-022-00349-3 ↗
93. Gong, S., Xie, T., Shao-Horn, Y., Gomez-Bombarelli, R., & Grossman, J. C. Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity. Preprint at https://arxiv.org/abs/2208.05039 (2022).10.21203/rs.3.rs-2042719/v1 ↗
94. Choudhary, K. Quantum computation for predicting electron and phonon properties of solids. J. Phys.: Cond. Matt. 33, 385501 (2021).
95. Nguyen, H. G. T. et al. A reference high-pressure co2 adsorption isotherm for ammonium zsm-5 zeolite: results of an interlaboratory study. Adsorption 24, 531–539 (2018).10.1007/s10450-018-9958-x ↗
96. IBM Quantum, https://quantum-computing.ibm.com (2021).
97. Ward, L. et al. Matminer: An open source toolkit for materials data mining. Comp. Mat. Sci. 152, 60–69 (2018).10.1016/j.commatsci.2018.05.018 ↗
98. Ke, G. et al. Lightgbm: A highly efficient gradient boosting decision tree. Adv. Neur. Info Proc. Sys. 30, 3146–3154 (2017).
99. Harrison, W. A. Electronic structure and the properties of solids: the physics of the chemical bond (Courier Corporation, 2012).
100. Garrity, K. F. & Choudhary, K. Database of wannier tight-binding hamiltonians using high-throughput density functional theory. Sci. Data 8, 106 (2021).10.1038/s41597-021-00885-z ↗