Expert-Grounded Automatic Prompt Engineering for Extracting Lattice Constants of High-Entropy Alloys from Scientific Publications Using Large Language Models
Source abstract
Abstract
Abstract Large language models (LLMs) have shown promise for scientific data extraction from publications but rely on manual prompt refinement. We present an expert-grounded automatic prompt optimization framework that enhances LLM entity extraction reliability. Using high-entropy alloy lattice constant extraction as a testbed, we optimized prompts for Claude 3.5 Sonnet through feedback cycles on seven expert-annotated publications. Despite a modest optimization budget, recall improved from 0.27 to > 0.9, demonstrating that a small, expert-annotated dataset can yield significant improvements. The optimized prompt was applied to extract lattice constants from 2267 publications, yielding data for 1861 compositions. The optimized prompt transferred effectively to newer models: Claude 4.5 Sonnet, GPT-5, and Gemini 2.5 Flash. Analysis revealed three categories of LLM failure modes: contextual hallucination, semantic misinterpretation, and unit conversion errors, emphasizing the need for validation protocols. These results establish feedback-guided prompt optimization as a low-cost, transferable methodology for reliable scientific data extraction, providing a scalable pathway for complex LLM-assisted research tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Computational and Text Analysis Methods · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (70)
Show source-provided references
1. Kim E, Huang K, Saunders A, McCallum A, Ceder G, Olivetti E (2017) Materials synthesis insights from scientific literature via text extraction and machine learning. Chem Mater 29(21):9436–944410.1021/acs.chemmater.7b03500 ↗
2. Venugopal V, Broderick SR, Rajan K (2019) A picture is worth a thousand words: applying natural language processing tools for creating a quantum materials database map. MRS Commun 9(4):1134–114110.1557/mrc.2019.136 ↗
3. Olivetti EA, Cole JM, Kim E, Kononova O, Ceder G, Han TY-J, Hiszpanski AM (2020) Data-driven materials research enabled by natural language processing and information extraction. Appl Phys Rev 7(4)10.1063/5.0021106 ↗
4. Shetty P, Ramprasad R (2021) Automated knowledge extraction from polymer literature using natural language processing. Iscience 24(1)10.1016/j.isci.2020.101922 ↗
5. Wang W, Jiang X, Tian S, Liu P, Dang D, Su Y, Lookman T, Xie J (2022) Automated pipeline for superalloy data by text mining. npj Comput Mater 8(1):910.1038/s41524-021-00687-2 ↗
6. Zheng Z, Zhang O, Borgs C, Chayes JT, Yaghi OM (2023) ChatGPT chemistry assistant for text mining and the prediction of MOF synthesis. J Am Chem Soc 145(32):18048–1806210.1021/jacs.3c05819 ↗
7. Gilligan LP, Cobelli M, Taufour V, Sanvito S (2023) A rule-free workflow for the automated generation of databases from scientific literature. npj Comput Mater 9(1):22210.1038/s41524-023-01171-9 ↗
8. Sayeed HM, Mohanty T, Sparks TD (2024) Annotating materials science text: a semi-automated approach for crafting outputs with gemini pro. Integr Mater Manuf Innov 13(2):445–45210.1007/s40192-024-00356-4 ↗
9. Dagdelen J, Dunn A, Lee S, Walker N, Rosen AS, Ceder G, Persson KA, Jain A (2024) Structured information extraction from scientific text with large language models. Nat Commun 15(1):141810.1038/s41467-024-45563-x ↗
10. Polak MP, Morgan D (2024) Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nat Commun 15(1):1569. https://doi.org/10.1038/s41467-024-45914-810.1038/s41467-024-45914-8 ↗
11. Jiang X, Wang W, Tian S, Wang H, Lookman T, Su Y (2025) Applications of natural language processing and large language models in materials discovery. npj Comput Mater 11(1):7910.1038/s41524-025-01554-0 ↗
12. Foppiano L, Lambard G, Amagasa T, Ishii M (2024) Mining experimental data from materials science literature with large language models: an evaluation study. Sci Technol Adv Mater Methods 4(1):2356506
13. Liu S, Wen T, Pattamatta AS, Srolovitz DJ (2024) A prompt-engineered large language model, deep learning workflow for materials classification. Mater Today 80:240–24910.1016/j.mattod.2024.08.028 ↗
14. Ekuma CE (2025) Dynamic in-context learning with conversational models for data extraction and materials property prediction. APL Mach Learn 3(1):01611910.1063/5.0254406 ↗
15. Biswajeet DD, Kadkhodaei S (2025) Leveraging large language models to address data scarcity in machine learning: applications in graphene synthesis. arXiv preprint arXiv:2503.0487010.2139/ssrn.6293971 ↗
16. Liu N, Jafarzadeh S, Lattimer BY, Ni S, Lua J, Yu Y (2025) Harnessing large language models for data-scarce learning of polymer properties. Nat Comput Sci 5(3):245–25410.1038/s43588-025-00768-y ↗
17. Turan I, Sparks T (2025) Revolutionizing scientific figure decoding: benchmarking LLM data extraction performance. ChemRxiv (2025) https://doi.org/10.26434/chemrxiv-2025-pcq8d10.26434/chemrxiv-2025-pcq8d ↗
18. Polak MP, Morgan D (2025) Leveraging vision capabilities of multimodal LLMs for automated data extraction from plots. arXiv preprint arXiv:2503.12326
19. Yanguas-Gil A (2026) Performance of AI agents based on reasoning language models on ALD process optimization tasks. arXiv preprint arXiv:2601.09980
20. White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, Elnashar A, Spencer-Smith J, Schmidt DC (2023) A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382
21. Schulhoff S, Ilie M, Balepur N, Kahadze K, Liu A, Si C, Li Y, Gupta A, Han H, Schulhoff S, et al (2024) The prompt report: a systematic survey of prompt engineering techniques. arXiv preprint arXiv:2406.06608
22. Giray L (2023) Prompt engineering with ChatGPT: a guide for academic writers. Ann Biomed Eng 51(12):2629–263310.1007/s10439-023-03272-4 ↗
23. Ha E, Ha SM, Gerelkhuu Z, Kim H-Y, Yoon TH (2025) AI-based nanotoxicity data extraction and prediction of nanotoxicity. Comput Struct Biotechnol J 29:138–14810.1016/j.csbj.2025.03.052 ↗
24. Zhang Y, Itani S, Khanal K, Okyere E, Smith G, Takahashi K, Zang J (2024) GPTArticleExtractor: an automated workflow for magnetic material database construction. J Magn Magn Mater 597:17200110.1016/j.jmmm.2024.172001 ↗
25. Gupta S, Mahmood A, Shetty P, Adeboye A, Ramprasad R (2024) Data extraction from polymer literature using large language models. Commun Mater 5(1):26910.1038/s43246-024-00708-9 ↗
26. Immanuel J, Mahata A (2025) Enhancing materials data workflows through object-oriented design and large language models. Integr Mater Manuf Innov 14:561–57210.1007/s40192-025-00431-4 ↗
27. Prasad D, Pimpude M, Alankar A (2024) Towards development of automated knowledge maps and databases for materials engineering using large language models. arXiv preprint arXiv:2402.11323
28. Yuksekgonul M, Bianchi F, Boen J, Liu S, Lu P, Huang Z, Guestrin C, Zou J (2025) Optimizing generative AI by backpropagating language model feedback. Nature 639(8055):609–61610.1038/s41586-025-08661-4 ↗
29. Miracle DB, Senkov ON (2017) A critical review of high entropy alloys and related concepts. Acta Mater 122:448–51110.1016/j.actamat.2016.08.081 ↗
30. George EP, Curtin WA, Tasan CC (2020) High entropy alloys: a focused review of mechanical properties and deformation mechanisms. Acta Mater 188:435–47410.1016/j.actamat.2019.12.015 ↗
31. Han L, Zhu S, Rao Z, Scheu C, Ponge D, Ludwig A, Zhang H, Gutfleisch O, Hahn H, Li Z et al (2024) Multifunctional high-entropy materials. Nat Rev Mater 9(12):846–86510.1038/s41578-024-00720-y ↗
32. Borg CK, Frey C, Moh J, Pollock TM, Gorsse S, Miracle DB, Senkov ON, Meredig B, Saal JE (2020) Expanded dataset of mechanical properties and observed phases of multi-principal element alloys. Sci Data 7(1):43010.1038/s41597-020-00768-9 ↗
33. Tandoc C, Qi L, Hu Y-J (2025) A bond-based model for accurate prediction of lattice parameters of BCC solid solution alloys. Materialia 40:10241010.1016/j.mtla.2025.102410 ↗
34. Maresca F, Curtin WA (2020) Mechanistic origin of high strength in refractory BCC high entropy alloys up to 1900K. Acta Mater 182:235–24910.1016/j.actamat.2019.10.015 ↗
35. Liu S, Lee K, Balachandran PV (2022) Integrating machine learning with mechanistic models for predicting the yield strength of high entropy alloys. J Appl Phys 132(10)10.1063/5.0106124 ↗
36. Li J, Fu X, Feng H, Liu B, Liaw PK, Fang Q (2023) Evaluating the solid solution, local chemical ordering, and precipitation strengthening contributions in multi-principal-element alloys. J Alloy Compd 938:16852110.1016/j.jallcom.2022.168521 ↗
37. Anthropic: Claude 3.5 Sonnet. https://www.anthropic.com/claude. Large language model (2024)
38. Feng R, Feng B, Gao MC, Zhang C, Neuefeind JC, Poplawsky JD, Ren Y, An K, Widom M, Liaw PK (2021) Superior high-temperature strength in a supersaturated refractory high-entropy alloy. Adv Mater 33(48):210240110.1002/adma.202102401 ↗
39. Lee K, Balachandran PV (2022) Explainable artificial intelligence approach for yield strength prediction in as-cast multi-principal element alloys. Materialia 26:10162810.1016/j.mtla.2022.101628 ↗
40. Karumuri S, Hernandez A, Mishra S, McClure Z, Tucker V, Flanagan JC, Hwang S, Sandhage KH, Bilionis I, Titus MS et al (2025) Design of high-hardness complex concentrated alloys from physics, machine learning, and experiments. J Appl Phys 138(8)10.1063/5.0274486 ↗
41. Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, Huang X, Zhao E, Zhang Y, Chen Y et al (2025) Siren’s song in the AI ocean: A survey on hallucination in Large Language Models. Comput Linguist 1–4610.1162/COLI.a.16 ↗
42. Huang L, Yu W, Ma W, Zhong W, Feng Z, Wang H, Chen Q, Peng W, Feng X, Qin B et al (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans Inf Syst 43(2):1–5510.1145/3703155 ↗
43. Wang X, Raj A, Luebbe M, Wen H, Xu S, Lu K (2025) Reliable end-to-end material information extraction from the literature with source-tracked multi-stage large language models. https://arxiv.org/abs/2510.05142
44. Manning CD, Raghavan P, Schütze H (2008) Introduction to information retrieval. Cambridge University Press, Cambridge10.1017/CBO9780511809071 ↗
45. Ward L, Agrawal A, Choudhary A, Wolverton C (2016) A general-purpose machine learning framework for predicting properties of inorganic materials. npj Comput Mater 2(1):1602810.1038/npjcompumats.2016.28 ↗
46. Wang J, Xu P, Ji X, Li M, Lu W (2023) Feature selection in machine learning for perovskite materials design and discovery. Materials 1610.3390/ma16083134 ↗
47. Sheikh S, Vela B, Attari V, Huang X, Morcos P, Hanagan J, Acemi C, Karaman I, Elwany A, Arroyave’ R (2024) Exploring chemistry and additive manufacturing design spaces: a perspective on computationally-guided design of printable alloys. Mater Res Lett 12:235–26310.1080/21663831.2024.2316204 ↗
48. Farrar EHE, Grayson MN (2022) Machine learning and semi-empirical calculations: a synergistic approach to rapid, accurate, and mechanism-based reaction barrier prediction. Chem Sci 13:7594–760310.1039/D2SC02925A ↗
49. Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–22210.1023/B:STCO.0000035301.49549.88 ↗
50. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F e1071: Misc functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.6-7. http://CRAN.R-project.org/package=e1071
51. Balachandran PV (2020) Adaptive machine learning for efficient materials design. MRS Bull 45(7):579–58610.1557/mrs.2020.163 ↗
52. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B Stat Methodol 58(1):267–28810.1111/j.2517-6161.1996.tb02080.x ↗
53. Holleis L, Shivaram BS, Balachandran PV (2019) Machine learning guided design of single-molecule magnets for magnetocaloric applications. Appl Phys Lett 114(22):222404. https://doi.org/10.1063/1.509455310.1063/1.5094553 ↗
54. Lin C-M, Juan C-C, Chang C-H, Tsai C-W, Yeh J-W (2015) Effect of Al addition on mechanical properties and microstructure of refractory $$\text{ Al}\rm x$$HfNbTaTiZr alloys. J Alloy Compd 624:100–107. https://doi.org/10.1016/j.jallcom.2014.11.06410.1016/j.jallcom.2014.11.064 ↗
55. Zuo TT, Li RB, Ren XJ, Zhang Y (2014) Effects of Al and Si addition on the structure and properties of CoFeNi equal atomic ratio alloy. J Magn Magn Mater 371:60–68. https://doi.org/10.1016/j.jmmm.2014.07.02310.1016/j.jmmm.2014.07.023 ↗
56. Zhu JM, Fu HM, Zhang HF, Wang AM, Li H, Hu ZQ (2010) Synthesis and properties of multiprincipal component AlCoCrFeNiSix alloys. Mater Sci Eng, A 527(27):7210–7214. https://doi.org/10.1016/j.msea.2010.07.04910.1016/j.msea.2010.07.049 ↗
57. Yang M, Huang C, Han J, Wu H, Zhao Y, Yang T, Jin S, Wang C, Li Z, Shu R, Wang C, Lu H, Sha G, Liu X (2023) Development of the high-strength ductile ferritic alloys via regulating the intragranular and grain boundary precipitation of G-phase. J Mater Sci Technol 136:180–199. https://doi.org/10.1016/j.jmst.2022.07.02910.1016/j.jmst.2022.07.029 ↗
58. Moon J, Tabachnikova E, Shumilin S, Hryhorova T, Estrin Y, Brechtl J, Liaw PK, Wang W, Dahmen KA, Zargaran A, Bae JW, Do H-S, Lee B-J, Kim HS (2021) Deformation behavior of a Co–Cr–Fe–Ni–Mo medium-entropy alloy at extremely low temperatures. Mater Today 50:55–68. https://doi.org/10.1016/j.mattod.2021.08.00110.1016/j.mattod.2021.08.001 ↗
59. Piorunek D, Frenzel J, Jöns N, Somsen C, Eggeler G (2020) Chemical complexity, microstructure and martensitic transformation in high entropy shape memory alloys. Intermetallics 122:106792. https://doi.org/10.1016/j.intermet.2020.10679210.1016/j.intermet.2020.106792 ↗
60. Jiang Y, Tang H, Li Z, Zheng D, Tu Y (2022) Additive manufactured Mg–Gd–Y–Zr alloys: effects of Gd content on microstructure evolution and mechanical properties. Addit Manuf 59:103136. https://doi.org/10.1016/j.addma.2022.10313610.1016/j.addma.2022.103136 ↗
61. Pan S, Zhang JQ, He BB, Huang MX (2022) High strain gradient induced nanograin in a cocrnivc medium entropy alloy with lamellar carbide. Mater Sci Eng, A 852:143692. https://doi.org/10.1016/j.msea.2022.14369210.1016/j.msea.2022.143692 ↗
62. Wu CL, Zhang S, Zhang CH, Zhang H, Dong SY (2017) Phase evolution and cavitation erosion–corrosion behavior of $$\text{ FeCoCrAlNiTi}x$$ high entropy alloy coatings on 304 stainless steel by laser surface alloying. J Alloy Compd 698:761–770. https://doi.org/10.1016/j.jallcom.2016.12.19610.1016/j.jallcom.2016.12.196 ↗
63. Zhang FX, Tong Y, Kirkham M, Huq A, Bei H, Weber WJ, Zhang Y (2020) Structural disorder, phase stability and compressibility of refractory body-centered cubic solid–solution alloys. J Alloy Compd 847:155970. https://doi.org/10.1016/j.jallcom.2020.15597010.1016/j.jallcom.2020.155970 ↗
64. Cui P, Ma Y, Zhang L, Zhang M, Fan J, Dong W, Yu P, Li G (2018) Microstructure and mechanical behaviors of $$\text{ CoFeNiMnTi}\rm x\text{ Al}\rm 1-x$$ high entropy alloys. Mater Sci Eng, A 731:124–130. https://doi.org/10.1016/j.msea.2018.06.02210.1016/j.msea.2018.06.022 ↗
65. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A et al (2022) Training language models to follow instructions with human feedback. Adv Neural Inf Process Syst 35:27730–27744
66. Zhang M, Liu S, Dong S, Yin M, Hu Y, Wang X, Ma S, Wang S, Indurthi SR, Deng H et al (2025) Complex logical instruction generation. arXiv preprint arXiv:2508.09125
67. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459–9474
68. Ma SG, Zhang Y (2012) Effect of Nb addition on the microstructure and properties of AlCoCrFeNi high-entropy alloy. Mater Sci Eng, A 532:480–486. https://doi.org/10.1016/j.msea.2011.10.11010.1016/j.msea.2011.10.110 ↗
69. Zhu JM, Fu HM, Zhang HF, Wang AM, Li H, Hu ZQ (2010) Microstructures and compressive properties of multicomponent $$\text{ AlCoCrFeNiMo}\rm x$$ alloys. Mater Sci Eng, A 527(26):6975–6979. https://doi.org/10.1016/j.msea.2010.07.02810.1016/j.msea.2010.07.028 ↗
70. Liu S, Booth TR, Ji Y, Reinhart W, Balachandran PV. Replication data For: expert-grounded prompt engineering for extracting lattice constants of high entropy alloys from scientific publications using large language models. https://doi.org/10.18130/V3/ABITMJ10.18130/V3/ABITMJ ↗