Developing a Hybrid Morphological Analyzer for Low-Resource Languages
Source abstract
Abstract
Morphological analysis is the fundamental and preliminary task for Natural Language Processing (NLP) applications, which involve speech and language. Kannada is a low-resource language belonging to the Dravidian language family, which is highly agglutinative and morphologically rich in nature, where dataset development is happening rapidly due to the increasing demands of NLP tools. This study presents a hybrid approach that integrates rule-based and Transformer-based techniques, aiming to maximize their strengths while minimizing the respective limitations. In the Kannada language, the analysis of inflections has been challenging due to morphological richness, and to address this issue, 85 paradigms are created using Lttoolbox of Apertium. Further, a Transformer model is trained with the generated nominal data to generate the morphological analysis for the out-of-vocabulary inflections. The hybrid approach can be easily extended to new words as they are added to the dictionary. The obtained results are on a test set for inflections in Kannada precision: 0.924; recall: 0.925; and F1 score: 0.925. The main contributions include rule extraction for paradigm design at the word level, morphological analysis for nouns, verbs, adjectives, pronouns, and indeclinables on a benchmark dataset and morphological analysis generation using the Transformer architecture.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Language and cultural evolution
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science & Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education (MAHE), Manipal 576104, Karnataka, India
Department of Liberal Arts, Humanities and Social Sciences (DLHS), Manipal Academy of Higher Education (MAHE), Manipal 576104, Karnataka, India
References (41)
Show source-provided references
1. Jurafsky, D., and Martin, J. (2009). Speech and Language Processing, Prentice-Hall, Inc.. [2nd ed.].
2. Khurana, D., Koli, A., Khatter, K., and Singh, S. (2017). Natural Language Processing: State of The Art, Current Trends and Challenges. arXiv.
3. Quirk, R., and Crystal, D. (2010). A Comprehensive Grammar of the English Language, Pearson. Available online: https://books.google.co.in/books?id=XnOdYhVoEZ0C.
4. Sridhar, S. (2007). Modern Kannada Grammar, Manohar. Available online: https://books.google.co.in/books?id=XvoVXcXV3X4C.
5. Wiemerslage, A., Silfverberg, M., Yang, C., McCarthy, A., Nicolai, G., Colunga, E., and Kann, K. (2022). Morphological Processing of Low-Resource Languages: Where We Are and What’s Next. *Findings of the Association for Computational Linguistics: ACL 2022*, Association for Computational Linguistics.10.18653/v1/2022.findings-acl.80 ↗
6. Pan, Y., Li, X., Yang, Y., and Dong, R. (2020). Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation. arXiv.
7. Tsai, M.-F., Nakov, P., and Ng, H.T. (2010). Morphological Analysis for Resource-Poor Machine Translation, Department of Computer Science, National University of Singapore. Technical Report TR22/10.
8. Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., and McClosky, D. (2014, January 23–24). The Stanford CoreNLP Natural Language Processing Toolkit. Proceedings of the 52nd Annual Meeting Of The Association For Computational Linguistics: System Demonstrations, Baltimore, MD, USA. Available online: https://aclanthology.org/P14-5010.10.3115/v1/P14-5010 ↗
9. Nair, J., Aiswarya, L., and Sruthy, P. (2021, January 23–24). A Study on Morphological Analyser for Indian Languages: A Literature Perspective. Proceedings of the Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India.10.1007/978-3-030-81462-5_11 ↗
10. Britannica, T. (2024, October 02). Encyclopedia Britannica-“Indian Languages”. Available online: https://www.britannica.com/topic/Indian-languages.
11. Rani, S. (2023, March 02). Department of Official Language, Ministry of Home Affairs, Government of India NDCC, Available online: https://rajbhasha.gov.in/en/languages-included-eighth-schedule-indian-constitution.
12. Gupta, R., Goyal, P., and Diwakar, S. (2010, January 6–8). Transliteration Among Indian Languages Using WX Notation. Proceedings of the KONVENS, Saarbrücken, Germany.
13. Krishnamurti, B. (2003). The Dravidian Languages, Cambridge University Press. Available online: https://books.google.co.in/books?id=54fV7Lwu3fMC.10.1017/CBO9780511486876 ↗
14. Kempegowda, D. (1997). Kannada Bhasha Swaroopa, Kuvempu Kannada Research Center.
15. A Rule based Kannada Morphological Analyzer and Generator using Finite State Transducer
16. Stump, G. (2002). Morphological and syntactic paradigms: Arguments for a theory of paradigm linkage. Yearbook of Morphology 2001, Springer.10.1007/978-94-017-3726-5_5 ↗
17. Jena, I., Chaudhury, S., Chaudhry, H., and Sharma, D. (2011). Developing Oriya Morphological Analyzer Using Lt-Toolbox. Information Systems For Indian Languages, Springer.10.1007/978-3-642-19403-0_20 ↗
18. A deep learning approach for Malayalam morphological analysis at character level10.1016/j.procs.2018.05.058 ↗
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
20. Goyal, V., and Lehal, G. (2008, January 16–18). Hindi Morphological Analyzer and Generator. Proceedings of the 2008 First International Conference on Emerging Trends in Engineering and Technology, Nagpur, India.10.1109/ICETET.2008.11 ↗
21. Malladi, D., and Mannem, P. (2013, January 14–19). Statistical Morphological Analyzer for Hindi. Proceedings of the Sixth International Joint Conference on Natural Language Processing, Nagoya, Japan. Available online: https://aclanthology.org/I13-1136.
22. Bapat, M., Gune, H., and Bhattacharyya, P. (2024, June 01). A Paradigm-Based Finite State Morphological Analyzer for Marathi. In Proceedings of the 1st Workshop on South and Southeast Asian Natural Language Processing, Beijing, China, 2010. Available online: https://aclanthology.org/W10-3604.
23. A Sequence Labeling Approach to Morphological Analyzer for Tamil Language
24. A Morphological Analyser for Malayalam—A Comparison of Different Apporaches
25. Prathibha, R., and Padma, M. (2013, January 20–21). Development of morphological analyzer for Kannada verbs. Proceedings of the Fifth International Conference on Advances in Recent Technologies in Communication and Computing (ARTCom 2013), Bangalore, India.10.1049/cp.2013.2219 ↗
26. Padma, M., and Prathibha, R. (2014). Development of Morphological Stemmer, Analyzer and Generator for Kannada Nouns. Emerging Research in Electronics, Computer Science and Technology, Springer.10.1007/978-81-322-1157-0_73 ↗
27. Morphological Analyzer for Kannada Inflectional Words Using Hybrid Approach
28. Kannada Morphological Analyser and Generator Using Trie
29. Murthy, K. (1999, January 1–4). A network and process model for morphological analysis/generation. Proceedings of the ICOSAL-2, the Second International Conference on South Asian Languages, Punjabi University, Patiala, India.
30. Advancing Morphological Analyser for Kannada Using NLP and Machine Learning Approaches
31. Şahin, K. (2024, May 07). Atlamaz TransMorpher: A Phonologically Informed Transformer-based Morphological Analyzer. ALTNLP, Available online: https://api.semanticscholar.org/CorpusID:259093155.
32. Dasari, P., Chelpuri, A., Vuppala, N., Marreddy, M., Krishnamurthy, P., and Mamidi, R. (2025, April 30). Transformer-Based Context Aware Morphological Analyzer for Telugu. In Proceedings of the Third Workshop on Speech and Language Technologies for Dravidian Languages, Varna, Bulgaria, 2023. Available online: https://aclanthology.org/2023.dravidianlangtech-1.4/.
33. Forcada, M., Boyan, Ivanov, Bonev, S., Ortiz, J., Sanchez, G., Martınez, F., Armentano-Oller, C., Montava, M., Tyers, F., and Rosell, M. (2021, September 11). Lttoolbox. Available online: https://wiki.apertium.org/w/images/d/d0/Apertium2-documentation.pdf.
34. (2022, August 01). V K Aravinda Sanka: ASCII Unicode Convertor. Available online: https://aravindavk.in/sanka/.
35. (2024, December 26). Wikipedia contributors Finite-State Machine—Wikipedia, The Free Encyclopedia. Available online: https://en.wikipedia.org/w/index.php?\title=Finite-state_machine&oldid=1262168421.
36. (2021, November 11). IIIT-Hyderabad Kannada Tree Bank, Kannada Annotated Dataset. Available online: https://ltrc.iiit.ac.in/showfile.php?filename=downloads/kolhi/.
37. Rai, D., and Peraje, D. (2013). Sankshipta Adhunika Kannada Nighantu (a Brief Lexicon of Modern Kannada), Saujanya Prasaranga.
38. Kulkarni, A. (2019). Sanskrit Parsing: Based on the Theories of Śābdabodha, Indian Institute of Advanced Study. Available online: https://books.google.co.in/books?id=sgFEygEACAAJ.
39. Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. (2017). Opennmt: Open-source toolkit for neural machine translation. arXiv.10.18653/v1/P17-4012 ↗
40. Klein, G., Hernandez, F., Nguyen, V., and Senellart, J. (2024, June 07). The OpenNMT Neural Machine Translation Toolkit: 2020 Edition. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), Virtual, 2020. Available online: https://aclanthology.org/2020.amta-research.9.
41. Walker, S. (2024, June 19). Perplexity in AI and NLP. Available online: https://klu.ai/glossary/perplexity.