Information Extraction from Multi-Domain Scientific Documents: Methods and Insights
Source abstract
Abstract
The rapid growth of scientific literature necessitates effective information extraction. However, existing methods face significant challenges, particularly when applied to multi-domain documents and low-resource languages. For Kazakh and Russian, there is a notable lack of annotated corpora and dedicated tools for scientific information extraction. To address this gap, we introduce SciMDIX (Scientific Multi-Domain Information extraction), a novel multi-domain dataset of scientific documents in Russian and Kazakh, annotated with entities and relations. Our study includes a comprehensive evaluation of entity recognition performance, comparing state-of-the-art models, such as BERT, LLaMA, GLiNER, and spaCy across four diverse domains (IT, Linguistics, Medicine, and Psychology) in both languages. The findings highlight the promise of spaCy and GLiNER for practical deployment in under-resourced language settings. Furthermore, we propose a new zero-shot relation extraction model that leverages a multimodal representation by integrating sentence context, entity mentions, and textual definitions of relation classes. Our model can predict semantic relations between entities in new documents, even for a language encountered during training. This capability is especially valuable for low-resource language scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Text Analysis Techniques · Semantic Web and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Information and Computational Technologies, Almaty 050010, Kazakhstan
A.P. Ershov Institute of Informatics Systems, Novosibirsk 630090, Russia
Global Education and Training, University of Illinois Urbana-Champaign, Champaign, IL 61801, USA
Department of Software Engineering, Satbayev University, Almaty 050013, Kazakhstan
Funding
Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan
References (27)
Show source-provided references
1. Liu, F., Vulić, I., Korhonen, A., and Collier, N. (2021, January 1–6). Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Online.10.18653/v1/2021.acl-short.72 ↗
2. Zhang, Q., Chen, Z., Pan, H., Caragea, C., Latecki, L.J., and Dragut, E.C. (2024, January 12–16). SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.726 ↗
3. Piskorski, J., Babych, B., Kancheva, Z., Kanishcheva, O., Lebedeva, M., Marcinczuk, M., Nakov, P., Osenova, P., Pivovarova, L., and Pollak, S. (2021, January 19–20). Slav-NER: The 3rd Cross-lingual Challenge on Recognition, Normalization, Classification, and Linking of Named Entities across Slavic languages. Proceedings of the Workshop on Balto-Slavic Natural Language Processing 2021, Kiev, Ukraine.
4. (2025, June 10). Linguistic Data Consortium. ACE 2005 Multilingual Training Corpus,, Available online: https://catalog.ldc.upenn.edu/LDC2006T06.
5. Zhang, Y., Zhong, V., Chen, D., Angeli, G., and Manning, C.D. (2017, January 9–11). Position-aware Attention and Supervised Data for Slot Filling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark.10.18653/v1/D17-1004 ↗
6. Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H. (November, January 31). Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1360 ↗
7. Bruches, E., Tikhobaeva, O., Dementyeva, Y., and Batura, T. (2022, January 12–17). TERMinator: A System for Scientific Texts Processing. Proceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Gyeongju, Republic of Korea.
8. NEREL: A Russian information extraction dataset with rich annotation for nested entities, relations, and wikidata entity links10.1007/s10579-023-09674-z ↗
9. Yeshpanov, R., Khassanov, Y., and Varol, H.A. (2022, January 20–25). KazNERD: Kazakh Named Entity Recognition Dataset. Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France.
10. Loukachevitch, N., Manandhar, S., Baral, E., Rozhkov, I., Braslavski, P., Ivanov, V., Batura, T., and Tutubalina, E. (2023). NEREL-BIO: A Dataset of Biomedical Abstracts Annotated with Nested Named Entities. Bioinformatics, 39.10.1093/bioinformatics/btad161 ↗
11. Haisa, G., and Altenbek, G. (2022). Deep Learning with Word Embedding Improves Kazakh Named-Entity Recognition. Information, 13.10.3390/info13040180 ↗
12. Tolegen, G., Toleu, A., and Mussabayev, R. (2020, January 11–12). Voted-Perceptron Approach for Kazakh Morphological Disambiguation. Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), Marseille, France.
13. Lafferty, J., McCallum, A., and Pereira, F.C. (July, January 28). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning (ICML), Williamstown, MA, USA.
14. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 3–5). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
15. Akhmetov, I., Aubakirov, S., Saparov, T., Mussabayev, R., Toleu, A., and Krassovitskiy, A. (2024, January 15–17). Machine Learning Methods for Kazakh Morphology: A Comprehensive Overview. Proceedings of the 2024 IEEE 3rd International Conference on Problems of Informatics, Electronics and Radio Engineering (PIERE), Novosibirsk, Russia.10.1109/PIERE62470.2024.10804966 ↗
16. Morphological parsing of Kazakh texts with deep learning approaches10.26577/JMMCS2024-v124-i4-a4 ↗
17. Named entity recognition for the kazakh language10.26577/JMMCS.2020.v107.i3.06 ↗
18. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
19. Interrater reliability: The kappa statistic10.11613/BM.2012.031 ↗
20. Kuratov, Y., and Arkhipov, M. (2019). Adaptation of deep bidirectional multilingual transformers for Russian language. arXiv.
21. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The llama 3 herd of models. arXiv.
22. Lora: Low-rank adaptation of large language models
23. Zaratiana, U., Tomeh, N., Holat, P., and Charnois, T. (2024, January 16–21). GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico.10.18653/v1/2024.naacl-long.300 ↗
24. Miranda, L.J., Kádár, Á., Boyd, A., Van Landeghem, S., Søgaard, A., and Honnibal, M. (2022). Multi hash embeddings in spaCy. arXiv.
25. Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. (2022). Text embeddings by weakly-supervised contrastive pre-training. arXiv.
26. Wang, S., Zhang, B., Xu, Y., Wu, Y., and Xiao, B. (2022, January 10–15). RCL: Relation Contrastive Learning for Zero-Shot Relation Extraction. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, USA.10.18653/v1/2022.findings-naacl.188 ↗
27. Wu, S., and He, Y. (2019, January 3–7). Enriching Pre-trained Language Model with Entity Information for Relation Classification. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, New York, NY, USA. CIKM’19.10.1145/3357384.3358119 ↗