SLM-MATRIX: a multi-agent trajectory reasoning and verification framework for enhancing language models in materials data extraction
Source abstract
Abstract
Small Language Models offer an efficient alternative for structured information extraction. We present SLM-MATRIX , a multi-path collaborative reasoning and verification framework based on SLMs, designed to extract material names, numerical values, and physical units from materials science literature. The framework integrates three complementary reasoning paths: a multi-agent collaborative path, a generator–discriminator path, and a dual cross-verification path. SLM-MATRIX achieves an accuracy of 92.85% on the BulkModulus dataset and reaches 77.68% accuracy on the MatSynTriplet dataset, both outperforming conventional methods and single-path models. Moreover, experiments on general reasoning benchmarks such as GSM8K and SVAMP validate the framework’s strong generalization capability. Ablation studies evaluate the effects of agent number, Mixture-of-Agents (MoA) depth, and discriminator design on overall performance. Overall, SLM-MATRIX presents an effective approach for high-quality material information extraction in resource-constrained and offers new insights into structured scientific text understanding tasks.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Materials Science · Semantic Web and Ontologies · Catalysis and Hydrodesulfurization Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (25)
Show source-provided references
1. Zhang, S. et al. OPT: Open Pre-trained Transformer Language Models. Preprint at https://doi.org/10.48550/arXiv.2205.01068 (2022).10.48550/arXiv.2205.01068 ↗
2. Chowdhery, A. et al. PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res. 24, 1–113 (2023).
3. Touvron, H. et al. LLaMA: Open and Efficient Foundation Language Models. Preprint at https://doi.org/10.48550/arXiv.2302.13971 (2023).10.48550/arXiv.2302.13971 ↗
4. Brown, T. et al. Language Models are Few-Shot Learners. in Advances in Neural Information Processing Systems vol. 33 1877–1901 (Curran Associates, Inc., 2020).
5. OpenAI et al. GPT-4 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2024).10.48550/arXiv.2303.08774 ↗
6. Ouyang, L. et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 35, 27730–27744 (2022).
7. Kim, E. et al. Materials Synthesis Insights from Scientific Literature via Text Extraction and Machine Learning. Chem. Mater. 29, 9436–9444 (2017).10.1021/acs.chemmater.7b03500 ↗
8. Mavračić, J., Court, C. J., Isazawa, T., Elliott, S. R. & Cole, J. M. ChemDataExtractor 2.0: Autopopulated Ontologies for Materials Science. J. Chem. Inf. Model. 61, 4280–4289 (2021).10.1021/acs.jcim.1c00446 ↗
9. Polak, M. P. & Morgan, D. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nat. Commun. 15, 1569 (2024).10.1038/s41467-024-45914-8 ↗
10. Vaswani, A. et al. Attention is All you Need. In Advances in Neural Information Processing Systems vol. 30 (Curran Associates, Inc., 2017).
11. Wang, X. et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In: The Eleventh International Conference on Learning Representations Preprint at https://doi.org/10.48550/arXiv.2203.11171 (2023).10.48550/arXiv.2203.11171 ↗
12. Gururangan, S. et al. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 8342–8360 (2020).10.18653/v1/2020.acl-main.740 ↗
13. Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 35, 24824–24837 (2022).
14. Valmeekam, K., Olmo, A., Sreedharan, S. & Kambhampati, S. Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). in (2022).
15. Alayrac, J.-B. et al. Flamingo: a Visual Language Model for Few-Shot Learning. Adv. Neural Inf. Process. Syst. 35, 23716–23736 (2022).
16. Wang, J., Wang, J., Athiwaratkun, B., Zhang, C. & Zou, J. Mixture-of-Agents Enhances Large Language Model Capabilities. in (2024).
17. Qi, Z. et al. Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers. CoRR Preprint at https://doi.org/10.48550/arXiv.2408.06195 (2024).10.48550/arXiv.2408.06195 ↗
18. Goodfellow, I. J. et al. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, vol. 27 (Curran Associates, Inc., 2014).
19. Mysore, S. et al. The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures. in Proceedings of the 13th Linguistic Annotation Workshop (eds. Friedrich, A., Zeyrek, D. & Hoek, J.) 56–64 (Association for Computational Linguistics, Florence, Italy). https://doi.org/10.18653/v1/W19-4007 (2019).10.18653/v1/W19-4007 ↗
20. Cobbe, K. et al. Training Verifiers to Solve Math Word Problems. Preprint at https://doi.org/10.48550/arXiv.2110.14168 (2021).10.48550/arXiv.2110.14168 ↗
21. Patel, A., Bhattamishra, S. & Goyal, N. Are NLP Models really able to Solve Simple Math Word Problems? In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies 2080–2094 (2021).10.18653/v1/2021.naacl-main.168 ↗
22. Gao, L. et al. PAL: Program-aided Language Models. in Proceedings of the 40th International Conference on Machine Learning 10764–10799 (PMLR, 2023).
23. Hendrycks, D. et al. Measuring Mathematical Problem Solving With the MATH Dataset. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) Preprint at https://doi.org/10.48550/arXiv.2103.03874 (2021).10.48550/arXiv.2103.03874 ↗
24. Yao, S. et al. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Advances in Neural Information Processing Systems 36, 11809–11822 (2023).
25. Hao, S. et al. Reasoning with Language Model is Planning with World Model. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing 8154−8173 (2023).10.18653/v1/2023.emnlp-main.507 ↗