Steel design based on a large language model
Source abstract
Abstract
The success of artificial intelligence (AI) in materials research heavily relies on the integrity of structured data and the construction of precise descriptors. In this study, we present an end-to-end pipeline from materials text to properties for steels based on a large language model. The objective is to enable quantitative predictions of properties with high-accuracy and explore new steels. The pipeline includes a materials language encoder, named SteelBERT, and a multimodal deep learning framework that maps the composition and text sequence of complex fabrication processes to mechanical properties. We demonstrate high accuracy on mechanical properties, including yield strength (YS), ultimate tensile strength (UTS), and elongation (EL) by predicting determination coefficients (R 2) reaching 78.17 % ( ± 3.40 %), 82.56 % ( ± 1.96 %), and 81.44 % ( ± 2.98 %) respectively. Further, through an additional fine-tuning strategy for the design of specific steels with small datasets, we show how the performance can be refined. With only 64 experimental samples of 15Cr austenitic stainless steels, we obtain an optimized model with R 2 of 89.85 % ( ± 6.17 %), 88.34 % ( ± 5.95 %) and 87.24 % ( ± 5.15 %) for YS, UTS and EL, that requires the user to input composition and text sequence for processing and which outputs mechanical properties. The model efficiently optimizes the text sequence for the fabrication process by suggesting a secondary round of cold rolling and tempering to yield an exceptional YS of 960 MPa, UTS of 1138 MPa, and EL of 32.5 %, exceeding those of reported 15Cr austenitic stainless steels.
Source: expertdata · Quality: candidate unverified · Source record ↗
Research context
OpenAlex topics: Machine Learning in Materials Science · Hydrogen embrittlement and corrosion behaviors in metals · Microstructure and Mechanical Properties of Steels
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (57)
Show source-provided references
1. Active learning guides discovery of a champion four-metal perovskite oxide for oxygen evolution electrocatalysis10.1038/s41563-023-01707-w ↗
2. Machine-learning-assisted materials discovery using failed experiments10.1038/nature17439 ↗
3. Machine learning–enabled high-entropy alloy discovery10.1126/science.abo4940 ↗
4. Inverse molecular design using machine learning: generative models for matter engineering10.1126/science.aat2663 ↗
5. Bond sensitive graph neural networks for predicting high temperature superconductors10.1002/mgea.48 ↗
6. Massively parallel pooled screening reveals genomic determinants of nanoparticle delivery10.1126/science.abm5551 ↗
7. Prospects of materials genome engineering frontiers10.1002/mgea.17 ↗
8. Open AI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F.L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H.W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S.P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S.S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Ł. Kaiser, A. Kamali, I. Kanitscheider, N.S. Keskar, T. Khan, L. Kilpatrick, J.W. Kim, C. Kim, Y. Kim, H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Ł. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C.M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S.M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F.de A.B. Peres, M. Petrov, H.P.de O. Pinto, Michael, Pokorny, M. Pokrass, V. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F.P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J.F.C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J.J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C.J. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, B. Zoph, GPT-4 Technical Report, (2023). http://arxiv.org/abs/2303.08774.
9. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, G. Penedo, The Falcon Series of Open Language Models, (2023). http://arxiv.org/abs/2311.1686.
10. BERT: pre-training of deep bidirectional transformers for language understanding
11. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, LLaMA: open and efficient foundation language models, (2023). https://doi.org/10.48550/arXiv.2302.13971.
12. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C.C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P.S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E.M. Smith, R. Subramanian, X.E. Tan, B. Tang, R. Taylor, A. Williams, J.X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom, Llama 2: open Foundation and Fine-Tuned Chat Models, (2023). http://arxiv.org/abs/2307.09288.
13. Palm: scaling language modeling with pathways
14. D. Driess, F. Xia, M.S.M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, P. Florence, PaLM-E: an Embodied Multimodal Language Model, (2023). http://arxiv.org/abs/2303.03378.
15. Attention is all you need
16. W.X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, J.-R. Wen, A Survey of Large Language Models, (2023). http://arxiv.org/abs/2303.18223.
17. J. Kaplan, S. McCandlish, T. Henighan, T.B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling Laws for Neural Language Models, (2020). http://arxiv.org/abs/2001.08361.
18. Pre-trained models: past, present and future10.1016/j.aiopen.2021.08.002 ↗
19. BatteryBERT: a pretrained language model for battery database enhancement10.1021/acs.jcim.2c00035 ↗
20. Advances in machine learning- and artificial intelligence-assisted material design of steels10.1007/s12613-022-2595-0 ↗
21. Mausam, MatSciBERT: a materials domain language model for text mining and information extraction10.1038/s41524-022-00784-w ↗
22. SciBERT: a pretrained language model for scientific text
23. polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics10.1038/s41467-023-39868-6 ↗
24. MaterialBERT for natural language processing of materials science texts
25. M. Chen, J. Tworek, H. Jun, Q. Yuan, H.P. de O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F.P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W.H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A.N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, W. Zaremba, Evaluating Large Language Models Trained on Code, (2021). http://arxiv.org/abs/2107.03374.
26. Language models are few-shot learners
27. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, C. Sutton, Program Synthesis with Large Language Models, (2021). http://arxiv.org/abs/2108.07732.
28. W.U. Ahmad, S. Chakraborty, B. Ray, K.-W. Chang, Unified Pre-training for Program Understanding and Generation, (2021). http://arxiv.org/abs/2103.06333.10.18653/v1/2021.naacl-main.211 ↗
29. Scaling deep learning for materials discovery10.1038/s41586-023-06735-9 ↗
30. An autonomous laboratory for the accelerated synthesis of novel materials10.1038/s41586-023-06734-w ↗
31. ChatGPT chemistry assistant for text mining and the prediction of MOF synthesis
32. Autonomous chemical research with large language models10.1038/s41586-023-06792-0 ↗
33. Third generation of advanced high-strength steels: processing routes and properties
34. Ductile 2-GPa steels with hierarchical substructure10.1126/science.add7857 ↗
35. Facile route to bulk ultrafine-grain steels for high strength and ductility10.1038/s41586-021-03246-3 ↗
36. Simultaneously enhancing strength-ductility synergy and strain hardenability via Si-alloying in medium-Al FeMnAlC lightweight steels10.1016/j.actamat.2022.118611 ↗
37. Study on the deformation mechanism of a high-nitrogen duplex stainless steel with excellent mechanical properties originated from bimodal grain design10.1016/j.actamat.2022.117670 ↗
38. Integrating data mining and machine learning to discover high-strength ductile titanium alloys10.1016/j.actamat.2020.10.056 ↗
39. On the use of transfer modeling to design new steels with excellent rotating bending fatigue resistance even in the case of very small calibration datasets10.1016/j.actamat.2022.118103 ↗
40. Dramatically enhanced combination of ultimate tensile strength and electric conductivity of alloys via machine learning screening10.1016/j.actamat.2020.09.068 ↗
41. Materials information extraction via automatically generated corpus10.1038/s41597-022-01492-2 ↗
42. Alloy synthesis and processing by semi-supervised text mining10.1038/s41524-023-01138-w ↗
43. Automated pipeline for superalloy data by text mining10.1038/s41524-021-00687-2 ↗
44. Deberta: decoding-enhanced bert with disentangled attention
45. Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing
46. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, ALBERT: a Lite BERT for Self-supervised Learning of Language Representations, (2020). http://arxiv.org/abs/1909.11942.
47. BART: denoising sequence-to-sequence pre-training for natural language generation
48. Optuna: a next-generation hyperparameter optimization framework
49. L. McInnes, J. Healy, J. Melville, UMAP: uniform Manifold Approximation and Projection for Dimension Reduction, (2020). http://arxiv.org/abs/1802.03426.
50. M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A.A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y.-C. Chen, Y.-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. Del Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R.J. Hewett, W. Hu, J. Huynh, D. Iter, S.A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y.J. Kim, L. Kurilenko, J.R. Lee, Y.T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C.C.T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L.L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, X. Zhou, Phi-3 technical report: a highly capable language model locally on your phone, (2024). http://arxiv.org/abs/2404.14219.
51. A.Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D.S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L.R. Lavaud, M.-A. Lachaux, P. Stock, T.L. Scao, T. Lavril, T. Wang, T. Lacroix, W.E. Sayed, Mistral 7B, (2023). http://arxiv.org/abs/2310.06825.
52. Neural machine translation of rare words with subword units
53. Japanese and Korean voice search
54. A new resource-saving, low chromium and low nickel duplex stainless steel 15Cr–xAl–2Ni–yMn10.1016/j.matdes.2013.06.043 ↗
55. Excellent combination of strength and ductility in 15Cr-2Ni duplex stainless steel based on ultrafine-grained austenite phase10.1016/j.msea.2017.02.056 ↗
56. Exploiting the non-equilibrium phase transformation in a 15Cr-2Ni-2Al-11Mn resource-saving duplex stainless steel10.1016/j.matdes.2016.10.076 ↗
57. Ferrite effects on the hydrogen embrittlement of 17-4PH stainless steel10.1108/ACMM-03-2022-2615 ↗