Fine-Tuning Large Language Models for Kazakh Text Simplification
Source abstract
Abstract
This paper addresses text simplification task for Kazakh, a morphologically rich, low-resource language, by introducing KazSim, an instruction-tuned model built on multilingual large language models (LLMs). First, we develop a heuristic pipeline to identify complex Kazakh sentences, manually validating its performance on 400 examples and comparing it against a purely LLM-based selection method; we then use this pipeline to assemble a parallel corpus of 8709 complex–simple pairs via LLM augmentation. For the simplification task, we benchmark KazSim against standard Seq2Seq systems, domain-adapted Kazakh LLMs, and zero-shot instruction-following models. On an automatically constructed test set, KazSim (Llama-3.3-70B) achieves BLEU 33.50, SARI 56.38, and F1 87.56 with a length ratio of 0.98, outperforming all baselines. We also explore prompt language (English vs. Kazakh) and conduct human evaluation with three native speakers: KazSim scores 4.08 for fluency, 4.09 for meaning preservation, and 4.42 for simplicity—significantly above GPT-4o-mini. Error analysis shows that remaining failures cluster into tone change, tense change, and semantic drift, reflecting Kazakh’s agglutinative morphology and flexible syntax.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Text Readability and Simplification · Natural Language Processing Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Information and Computational Technologies, Almaty 050010, Kazakhstan
AI Research Laboratory, Satbayev University, Almaty 050000, Kazakhstan
Faculty of Information Technology, Al-Farabi Kazakh National University, Almaty 050040, Kazakhstan
Funding
Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan
References (22)
Show source-provided references
1. Tolegen, G., Toleu, A., Mussabayev, R., Zhumazhanov, B., and Ziyatbekova, G. (2023, January 14–22). Generative Pre-Trained Transformer for Kazakh Text Generation Tasks. Proceedings of the 2023 19th International Asian School-Seminar on Optimization Problems of Complex Systems (OPCS), Novosibirsk, Moscow, Russian.10.1109/OPCS59592.2023.10275765 ↗
2. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
3. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and Efficient Foundation Language Models. arXiv.
4. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., and Wei, H. (2025). Qwen2.5 Technical Report. arXiv.
5. Ormaechea, L., Tsourakis, N., Schwab, D., Bouillon, P., and Lecouteux, B. (2023, January 16–17). Simple, Simpler and Beyond: A Fine-Tuning BERT-Based Approach to Enhance Sentence Complexity Assessment for Text Simplification. Proceedings of the 6th International Conference on Natural Language and Speech Processing (ICNLSP 2023), Online.
6. Mallinson, J., Sennrich, R., and Lapata, M. (2020, January 16–20). Zero-Shot Crosslingual Sentence Simplification. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.415 ↗
7. Kew, T., Chi, A., Vásquez-Rodríguez, L., Agrawal, S., Aumiller, D., Alva-Manchego, F., and Shardlow, M. (2023, January 6–10). BLESS: Benchmarking Large Language Models on Sentence Simplification. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.821 ↗
8. Miller, G.A. (1994, January 8–11). WordNet: A Lexical Database for English. Proceedings of the Workshop on Human Language Technology, Plainsboro, NJ, USA.10.3115/1075812.1075938 ↗
9. Shardlow, M. (2014, January 26–31). Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
10. Drndarević, B., and Saggion, H. (2012, January 7). Towards Automatic Lexical Simplification in Spanish: An Empirical Study. Proceedings of the First Workshop on Predicting and Improving Text Readability for Target Reader Populations, Montréal, QC, Canada.
11. Shardlow, M. (2013, January 4–9). A Comparison of Techniques to Automatically Identify Complex Words. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics Proceedings of the Student Research Workshop, Sofia, Bulgaria.
12. Carroll, J., Minnen, G., Canning, Y., Devlin, S., and Tait, J. (1998, January 26–30). Practical simplification of English newspaper text to assist aphasic readers. Proceedings of the AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology, Madison, WI, USA.
13. Chandrasekar, R., Doran, C., and Srinivas, B. (1996). Motivations and Methods for Text Simplification. COLING 1996 Volume 2, Proceedings of the 16th International Conference on Computational Linguistics, Copenhagen, Denmark, 5–9 August 1996, Association for Computational Linguistics.10.3115/993268.993361 ↗
14. Siddharthan, A. (2011, January 28–30). Text Simplification using Typed Dependencies: A Comparison of the Robustness of Different Generation Strategies. Proceedings of the 13th European Workshop on Natural Language Generation, Nancy, France.
15. Woodsend, K., and Lapata, M. (2011, January 27–29). Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, UK.
16. Zhang, X., and Lapata, M. (2017, January 9–11). Sentence Simplification with Deep Reinforcement Learning. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1062 ↗
17. Ryan, M.J., Naous, T., and Xu, W. (2023, January 9–14). Revisiting non-English Text Simplification: A Unified Multilingual Benchmark. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.269 ↗
18. Tolegen, G., Toleu, A., and Mussabayev, R. (2022, January 14–16). A Finite State Transducer Based Morphological Analyzer for The Kazakh Language. Proceedings of the 2022 7th International Conference on Computer Science and Engineering (UBMK), Diyarbakir, Turkey.10.1109/UBMK55850.2022.9919445 ↗
19. Toleu, A., Tolegen, G., and Mussabayev, R. (2022, January 12–17). Language-Independent Approach for Morphological Disambiguation. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea.
20. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
21. Koto, F., Joshi, R., Mukhituly, N., Wang, Y., Xie, Z., Pal, R., Orel, D., Mullah, P., Turmakhan, D., and Goloburda, M. (2025). Llama-3.1-Sherkala-8B-Chat: An Open Large Language Model for Kazakh. arXiv.
22. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). BERTScore: Evaluating Text Generation with BERT. arXiv.