Agent-based multimodal information extraction for nanomaterials
Source abstract
Abstract
Abstract Automating structured data extraction from scientific literature is a critical challenge with broad implications across domains. We introduce nanoMINER, a multi-agent system combining large language models and multimodal analysis to extract essential information from scientific research articles on nanomaterials. This system processes documents end-to-end, utilizing tools such as YOLO for visual data extraction and GPT-4o for linking textual and visual information. At its core, the ReAct agent orchestrates specialized agents to ensure comprehensive data extraction. We demonstrate the efficacy of the system by automating the assembly of nanomaterial and nanozyme datasets previously manually curated by domain experts. NanoMINER achieves high precision in extracting nanomaterial properties like chemical formulas, crystal systems, and surface characteristics. For nanozymes, we obtain near-perfect precision (0.98) for kinetic parameters and essential features such as Cmin and Cmax. To benchmark the system performance, we also compare nanoMINER to several baseline LLMs, including the most recent multimodal GPT-4.1, and show consistently higher extraction precision and recall. Our approach is extensible to other domains of materials science and fields like biomedicine, advancing data-driven research methodologies and automated knowledge extraction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Advanced Chemical Sensor Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (44)
Show source-provided references
1. Foppiano, L., Lambard, G., Amagasa, T. & Ishii, M. Mining experimental data from materials science literature with large language models: an evaluation study. Sci. Technol. Adv. Mater. Methods 4, 2356506 (2024).
2. Fine-tune BERT with Sparse Self-Attention Mechanism - ACL Anthology. https://aclanthology.org/D19-1361/ (accessed 2024-06-20).
3. Guo, M.-H. et al. Attention mechanisms in computer vision: a survey. Comput. Vis. Media 8, 331–368 (2022).10.1007/s41095-022-0271-y ↗
4. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A. Self-Attention Generative Adversarial Networks. In Proceedings of the 36th International Conference on Machine Learning; PMLR, 2019; 7354–7363.
5. Li, W., Qi, F., Tang, M. & Yu, Z. Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification. Neurocomputing 387, 63–77 (2020).10.1016/j.neucom.2020.01.006 ↗
6. Peng, R., Liu, K., Yang, P., Yuan, Z., Li, S. Embedding-based retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Data. arXiv 2023. https://doi.org/10.48550/ARXIV.2308.03107.10.48550/ARXIV.2308.03107 ↗
7. Patiny, L., Godin, G. Automatic Extraction of FAIR Data from Publications Using LLM. ChemRxiv December 4. https://doi.org/10.26434/chemrxiv-2023-05v1b-v2. (2023)10.26434/chemrxiv-2023-05v1b-v2 ↗
8. Biswas, A. & Talukdar, W. Robustness of structured data extraction from in-plane rotated documents using multi-modal Large Language Models (LLM). J. Artif. Intell. Res. 4, 176–195 (2024).
9. Jiang, A. Q. et al. Mistral 7B. arXiv. https://doi.org/10.48550/ARXIV.2310.06825. (2023)10.48550/ARXIV.2310.06825 ↗
10. Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/ (accessed 2024-06-20).
11. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. & Sutskever, I. Language models are unsupervised multitask learners. OpenAI Tech. Rep. (2019).
12. Evaluating GPT-4V (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs | Radiology. https://doi.org/10.1148/radiol.233270 (accessed 2024-06-20).10.1148/radiol.233270 ↗
13. Sun, Q., Cui, X., Zhou, W., Li, H. Exploiting GPT-4 Vision for Zero-Shot Point Cloud Understanding. arXiv January 15, 2024. https://doi.org/10.48550/arXiv.2401.07572.10.48550/arXiv.2401.07572 ↗
14. OpenAI. GPT-4o System Card. August 8, 2024. https://cdn.openai.com/gpt-4o-system-card.pdf.
15. Wu, Y., Hu, X., Fu, Z., Zhou, S., Li, J. GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding. arXiv June 14, 2024. https://doi.org/10.48550/arXiv.2406.09781.10.48550/arXiv.2406.09781 ↗
16. Schilling-Wilhelmi, M. et al. From text to insight: large language models for materials science data extraction. arXiv July 23, 2024. http://arxiv.org/abs/2407.16867 (accessed 2024-09-09).
17. Wang, X., Huey, S. L., Sheng, R., Mehta, S., Wang, F. SciDaSynth: Interactive structured knowledge extraction and synthesis from scientific literature with large language model. arXiv April 21, 2024. http://arxiv.org/abs/2404.13765 (accessed 2024-09-09).
18. Dagli, M. M. et al. Development and validation of a novel AI framework using NLP with LLM integration for relevant clinical data extraction through automated chart review. Sci. Rep. 14, 26783 (2024).10.1038/s41598-024-77535-y ↗
19. Autonomous Discovery in the Chemical Sciences Part II: Outlook - Coley - 2020 - Angewandte Chemie International Edition - Wiley Online Library. https://doi.org/10.1002/anie.201909989 (accessed 2024-06-20).10.1002/anie.201909989 ↗
20. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS Central Science. https://doi.org/10.1021/acscentsci.9b00576 (accessed 2024-06-20).10.1021/acscentsci.9b00576 ↗
21. The CHEMDNER corpus of chemicals and drugs and its annotation principles. J. Cheminform. | Full Text. https://doi.org/10.1186/1758-2946-7-S1-S2 (accessed 2024-06-20).10.1186/1758-2946-7-S1-S2 ↗
22. Dagdelen, J. et al. Structured information extraction from scientific text with large language models. Nat. Commun. 15, 1418 (2024).10.1038/s41467-024-45563-x ↗
23. Sun, L. et al. ChatGPT combining machine learning for the prediction of nanozyme catalytic types and activities. J. Chem. Inf. Model. 2024, acs.jcim.4c00600. https://doi.org/10.1021/acs.jcim.4c00600.10.1021/acs.jcim.4c00600 ↗
24. Lee, S. et al. Data-driven analysis of text-mined seed-mediated syntheses of gold nanoparticles. Digit. Discov. 4, 93–104 (2025).10.1039/D4DD00158C ↗
25. Ansari, M. & Moosavi, S. M. Agent-based learning of materials datasets from the scientific literature. Digit. Discov. 3, 2607–2617 (2024).10.1039/D4DD00252K ↗
26. Solovev, G. V. et al. Towards LLM-driven multi-agent pipeline for drug discovery: neurodegenerative diseases case study. In Int. Conf. Learn. Represent. (ICLR) (2024).
27. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: Las Vegas, NV, USA, 2016; 779–788. https://doi.org/10.1109/CVPR.2016.91.10.1109/CVPR.2016.91 ↗
28. Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv 2022. https://doi.org/10.48550/ARXIV.2210.03629.10.48550/ARXIV.2210.03629 ↗
29. Comprehensive Examination of Instruction-Based Language Models: A Comparative Analysis of Mistral-7B and Llama-2-7B|IEEE Conference Publication | IEEE Xplore. https://ieeexplore.ieee.org/abstract/document/10434081 (accessed 2024-06-20).
30. Razlivina, J., Dmitrenko, A. & Vinogradov, V. AI-powered knowledge base enables transparent prediction of nanozyme multiple catalytic activity. J. Phys. Chem. Lett. 15, 5804–5813 (2024).10.1021/acs.jpclett.4c00959 ↗
31. Chataut, S. et al. Comparative study of domain driven terms extraction using large language models. arXiv 2024. https://doi.org/10.48550/ARXIV.2404.02330.10.48550/ARXIV.2404.02330 ↗
32. Mimicking horseradish peroxidase and oxidase using ruthenium nanomaterials - RSC Advances (RSC Publishing). https://pubs.rsc.org/en/content/articlelanding/2017/ra/c7ra10370k.10.1039/C7RA10370K ↗
33. Hiroki Nakayama and; Takahiro Kubo and; Junya Kamura and; Yasufumi Taniguchi and; Xu Liang. doccano: Text Annotation Tool for Human. doccano. https://doccano.web.webis.de/ (accessed 2024-06-20).
34. Decker, J. pdfplumber: Plumb a PDF for detailed information about each char, rectangle, line, etc. GitHub. https://github.com/jsvine/pdfplumber (2023).
35. Papers with Code - PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition. https://paperswithcode.com/paper/padellm-ner-parallel-decoding-in-large (accessed 2024-06-20).
36. Paolini, G. et al. Structured prediction as translation between augmented natural languages. In Int. Conf. Learn. Represent. (ICLR) 1–20 (2021).
37. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv May 24, 2019. https://doi.org/10.48550/arXiv.1810.04805.10.48550/arXiv.1810.04805 ↗
38. Watanabe, S. Tree-structured parzen estimator: understanding its algorithm components and their roles for better empirical performance. arXiv May 26, 2023. http://arxiv.org/abs/2304.11127 (accessed 2024-06-20).
39. Ott, M. et al. Fairseq: A fast, extensible toolkit for sequence modeling. arXiv April 1, 2019. https://doi.org/10.48550/arXiv.1904.01038.10.48550/arXiv.1904.01038 ↗
40. Popel, M. & Bojar, O. Training Tips for the Transformer Model. Prague Bull. Math. Linguist. 110, 43–70 (2018).10.2478/pralin-2018-0002 ↗
41. Loshchilov, I., Hutter, F. Decoupled weight decay regularization. arXiv January 4, 2019. https://doi.org/10.48550/arXiv.1711.05101.10.48550/arXiv.1711.05101 ↗
42. Shazeer, N., Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv April 11, 2018. https://doi.org/10.48550/arXiv.1804.04235.10.48550/arXiv.1804.04235 ↗
43. Pokrass, M. Introducing Structured Outputs in the API. https://openai.com/index/introducing-structured-outputs-in-the-api/ (accessed 2024-08-10).
44. Hancock, J. M. Jaccard Distance (Jaccard Index, Jaccard Similarity Coefficient). In Dictionary of Bioinformatics and Computational Biology; Hancock, J. M., Zvelebil, M. J., Eds.; Wiley, 2004. https://doi.org/10.1002/9780471650126.dob0956.10.1002/9780471650126.dob0956 ↗