An Analysis of the Training Data Impact for Domain-Adapted Tokenizer Performances—The Case of Serbian Legal Domain Adaptation
Source abstract
Abstract
Various areas of natural language processing (NLP) have greatly benefited from the development of large language models in recent years. This research addresses the challenge of developing efficient tokenizers for transformer-based domain-specific language models. Tokenization efficiency within transformer-based models is directly related to model efficiency, which motivated the research we present in this paper. Our goal in this research was to demonstrate that the appropriate selection of data used for tokenizer training has a significant impact on tokenizer performance. Subsequently, we will demonstrate that efficient tokenizers and models can be developed even if language resources are limited. To do so, we will present a domain-adapted large language model tokenizer developed for masked language modeling of the Serbian legal domain. In this paper, we will present a comparison of the tokenization performance for a domain-adapted tokenizer in version 2 of the SrBERTa language model we developed, against the performances of five other tokenizers belonging to state-of-the-art multilingual, Slavic or Serbian-specific models—XLM-RoBERTa (base-sized), BERTić, Jerteh-81, SrBERTa v1, NER4Legal_SRB. The comparison is performed using a test dataset consisting of 275,660 samples of legal texts written in the Cyrillic alphabet gathered from the Official Gazette of the Republic of Serbia. This dataset contains 197,134 distinct words, while the overall word count is 5,265,352. We will show that our tokenizer, trained upon a domain-adapted dataset, outperforms presented tokenizers by at least 4.5% ranging to 54.62%, regarding the number of tokens generated for the whole test dataset. In terms of tokenizer fertility, we will show that our tokenizer outperforms compared tokenizers by at least 6.39% ranging to 56.8%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electronic Engineering, University of Nis, 18000 Nis, Serbia
Funding
Ministry of Science, Technological Development and Innovation of the Republic of Serbia
References (56)
Show source-provided references
1. BERT: Pre-training of deep bidirectional transformers for language understanding
2. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. arXiv.
3. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6–12). Language models are few-shot learners. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Vancouver, BC, Canada.
4. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., and Grella, M. (2023). Rwkv: Reinventing rnns for the transformer era. arXiv.10.18653/v1/2023.findings-emnlp.936 ↗
5. Bengio, Y., Ducharme, R., and Vincent, P. (2023, January 10–16). A neural probabilistic language model.Part of Advances in neural information processing systems. Proceedings of the 2000 Neural Information Processing Systems (NIPS) Conference, Denver, CO, USA.
6. Schwenk, H., Dechelotte, D., and Gauvain, J.-L. (2006, January 17–21). Continuous space language models for statistical machine translation. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, Sydney, Australia.10.3115/1273073.1273166 ↗
7. Mikolov, T., Deoras, A., Povey, D., Burget, L., and Černocky, J. (2011, January 11–15). Strategies for training large scale neural network language models. Proceedings of the 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, Waikoloa, HI, USA.10.1109/ASRU.2011.6163930 ↗
8. Deep learning–based text classification: A comprehensive review10.1145/3439726 ↗
9. Sutskever, I., Vinyals, O., and Le, Q.V. Sequence to sequence learning with neural networks. Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 (NIPS’14), Vol. 2. MIT Press, Cambridge, MA, USA.
10. Cho, K., Van Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv.10.3115/v1/W14-4012 ↗
11. Language Models are Unsupervised Multitask Learners
12. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020, January 5–10). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online.10.18653/v1/2020.acl-main.703 ↗
13. Goldman, O., Caciularu, A., Eyal, M., Cao, K., Szpektor, I., and Tsarfaty, R. (2024). Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance. arXiv.10.18653/v1/2024.findings-acl.134 ↗
14. Kong, Z., Li, Y., Zeng, F., Xin, L., Messica, S., Lin, X., Zhao, P., Kellis, M., Tang, H., and Zitnik, M. (2025). Token Reduction Should Go Beyond Efficiency in Generative Models—From Vision, Language to Multimodality. arXiv.
15. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020, January 5–10). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.747 ↗
16. Ljubešić, N., and Lauc, D. (2021, January 20). BERTić—The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian. Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, Online.
17. Škorić, M. (2024). Novi jezički modeli za srpski jezik. arXiv, (In Serbian).
18. Bogdanović, M., Kocić, J., and Stoimenov, L. (2024). SRBerta—A Transformer Language Model for Serbian Cyrillic Legal Texts. Information, 15.10.3390/info15020074 ↗
19. Kalušev, V., and Brkljač, B. (2025). Named entity recognition for Serbian legal documents: Design, methodology and dataset development. arXiv.
20. Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., and Brahma, S. (2022). Scaling instruction-finetuned language models. arXiv.
21. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. (2022). Codegen: An open large language model for code with multi-turn program synthesis. arXiv.
22. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T.L., Bari, M.S., Shen, S., Yong, Z.X., and Schoelkopf, H. (2022). Crosslingual generalization through multitask finetuning. arXiv.10.18653/v1/2023.acl-long.891 ↗
23. Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., and Zhang, X. (2021). Pangu-α: Large-scale autoregressive pretrained chinese language models with auto parallel computation. arXiv.
24. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., and Azhar, F. (2022). Llama: Open and efficient foundation language models. arXiv.
25. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, E., Heslow, D., Launay, J., and Malartic, Q. (2025, May 21). Falcon-40B: An Open Large Language Model with State-of-the-Art Performance. Available online: https://huggingface.co/tiiuae/falcon-40b.
26. Gu, A., and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv.
27. Facebook (2022, May 25). facebook/opt-125m. Available online: https://huggingface.co/facebook/opt-125m.
28. Facebook (2022, November 22). facebook/galactica-125m. Available online: https://huggingface.co/facebook/galactica-125m.
29. BigScience (2022, November 22). bigscience/bloom-560m. Available online: https://huggingface.co/bigscience/bloom-560m.
30. Bisk, Y., Zellers, R., Le Bras, R., Gao, J., and Choi, Y. (2020, January 7–12). Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6239 ↗
31. Microsoft (2023, December 10). microsoft/phi-2. Available online: https://huggingface.co/microsoft/phi-2.
32. Microsoft (2025, June 25). microsoft/phi-3-mini. Available online: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct.
33. (2023, December 10). Tinyllama. Available online: https://huggingface.co/tinyllama.
34. Meituan (2025, June 25). Mobilellama. Available online: https://huggingface.co/mtgv/MobileLLaMA-1.4B-Base.
35. Alibaba (2024, September 14). Qwen 2.5. Available online: https://qwenlm.github.io/blog/qwen2.5/.
36. Google (2025, June 25). Gemma. Available online: https://huggingface.co/google/gemma-3-4b-it.
37. Apple (2024, April 28). Openelm. Available online: https://huggingface.co/apple/OpenELM.
38. HuggingFace (2025, June 25). Smollm. Available online: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct.
39. DataBricks (2023, April 28). databricks/dolly-v2-3b. Available online: https://huggingface.co/databricks/dolly-v2-3b.
40. H2O.ai (2025, January 10). h2o-danube3-4b-base. Available online: https://huggingface.co/h2oai/h2o-danube3-4b-base.
41. Padiu, B., Iacob, R., Rebedea, T., and Dascalu, M. (2024). To What Extent Have LLMs Reshaped the Legal Domain So Far? A Scoping Literature Review. Information, 15.10.37766/inplasy2024.9.0055 ↗
42. Tagarelli, A., and Simeri, A. (2022). LamBERTa: Law Article Mining Based on Bert Architecture for the Italian Civil Code. IRCDL, Springer.
43. Exploring LLMs Applications in Law: A Literature Review on Current Legal NLP Approaches10.1109/ACCESS.2025.3533217 ↗
44. Mellinkoff, D. (2004). The Language of the Law, Wipf and Stock Publishers.
45. Mertz, E. (2007). The Language of Law School: Learning to “Think Like a Lawyer”, Oxford University Press.10.1093/acprof:oso/9780195183108.001.0001 ↗
46. Tiersma, P.M. (1999). Legal Language, University of Chicago Press.
47. BERT syntactic transfer: A computational experiment on Italian, French and English languages10.1016/j.csl.2021.101261 ↗
48. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing10.1145/3458754 ↗
49. Sachidananda, V., Kessler, J., and Lai, Y.A. (2021, January 10). Efficient Domain Adaptation of Language Models via Adaptive Tokenization. Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, Virtual.10.18653/v1/2021.sustainlp-1.16 ↗
50. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N.A. (2020, January 5–10). Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual.10.18653/v1/2020.acl-main.740 ↗
51. Dagan, G., Synnaeve, G., and Rozière, B. (2024). Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv.
52. Occiglot (2025, May 22). EU Tokenizer Performance. Available online: https://occiglot.eu/posts/eu_tokenizer_perfomance/.
53. (2025, May 26). Byte-Pair Encoding tokenization. Available online: https://huggingface.co/learn/llm-course/en/chapter6/5.
54. (2025, May 26). Summary of the Tokenizers. Available online: https://huggingface.co/docs/transformers/tokenizer_summary#byte-pairencoding.
55. Bogdanović, M., Frtunić Gligorijević, M., Kocić, J., and Stoimenov, L. (2025). Improving Text Recognition Accuracy for Serbian Legal Documents Using BERT. Appl. Sci., 15.10.3390/app15020615 ↗
56. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv.