Better Word Representation Vectors Using Syllabic Alphabet: A Case Study of Swahili
Source abstract
Abstract
Deep learning has extensively been used in natural language processing with sub-word representation vectors playing a critical role. However, this cannot be said of Swahili, which is a low resource and widely spoken language in East and Central Africa. This study proposed novel word embeddings from syllable embeddings (WEFSE) for Swahili to address the concern of word representation for agglutinative and syllabic-based languages. Inspired by the learning methodology of Swahili in beginner classes, we encoded respective syllables instead of characters, character n-grams or morphemes of words and generated quality word embeddings using a convolutional neural network. The quality of WEFSE was demonstrated by the state-of-art results in the syllable-aware language model on both the small dataset (31.229 perplexity value) and the medium dataset (45.859 perplexity value), outperforming character-aware language models. We further evaluated the word embeddings using word analogy task. To the best of our knowledge, syllabic alphabets have not been used to compose the word representation vectors. Therefore, the main contributions of the study are a syllabic alphabet, WEFSE, a syllabic-aware language model and a word analogy dataset for Swahili.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Software Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, West Hi-Tech Zone, Chengdu 611731, China
School of Information and Communication Technology, South Eastern Kenya University, Kitui 170-90200, Kenya
References (60)
Show source-provided references
1. Recent trends in deep learning based natural language processing10.1109/MCI.2018.2840738 ↗
2. A logical calculus of the ideas immanent in nervous activity10.1007/BF02478259 ↗
3. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. (2010, January 26–30). Recurrent neural network based language model. Proceedings of the Eleventh Annual Conference of the International Speech Communication Association, Chiba, Japan.10.21437/Interspeech.2010-343 ↗
4. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, Inc.
5. Deep learning10.1038/nature14539 ↗
6. Collobert, R., and Weston, J. (2008, January 5–9). A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.10.1145/1390156.1390177 ↗
7. Botha, J., and Blunsom, P. (, January Beijing,). Compositional morphology for word representations and language modelling. Proceedings of the International Conference on Machine Learning, 21–26 January 2014.
8. Ling, W., Dyer, C., Black, A.W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., and Luís, T. (2015, January 17–21). Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1176 ↗
9. Mikolov, T., Sutskever, I., Deoras, A., Le, H.S., Kombrink, S., and Cernocky, J. (2012). Subword lAnguage Modeling with Neural Networks, Faculty of Information Technology, Brno University of Technology.
10. Turian, J., Ratinov, L., and Bengio, Y. (2010, January 11–16). Word representations: a simple and general method for semi-supervised learning. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden.
11. Kim, Y., Jernite, Y., Sontag, D., and Rush, A.M. (2016, January 12–17). Character-aware neural language models. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10362 ↗
12. Chen, X., Xu, L., Liu, Z., Sun, M., and Luan, H. (2015, January 25–31). Joint learning of character and word embeddings. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
13. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
14. Lazaridou, A., Marelli, M., Zamparelli, R., and Baroni, M. (2013, January 4–9). Compositional-ly derived representations of morphologically complex words in distributional semantics. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria.
15. Cao, K., and Rei, M. (2016, January 11). A Joint Model for Word Embedding and Word Morphology. Proceedings of the 1st Workshop on Representation Learning for NLP, Berlin, Germany.10.18653/v1/W16-1603 ↗
16. Thomas, S., Seltzer, M.L., Church, K., and Hermansky, H. (2013, January 26–31). Deep neural network features and semi-supervised training for low resource speech recognition. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638959 ↗
17. De Pauw, G., Wagacha, P.W., and de Schryver, G.M. (2011). Towards English-Swahili Machine Translation, Research Workshop of the Israel Science Foundation.
18. Williams, E. (2014). Bridges and Barriers: Language in African Education and Development, Routledge.10.4324/9781315759463 ↗
19. A Morphological Classification of Kiswahili
20. Indakwa, J., and Ballali, D. (1995). Beginner’s Swahili, Hippocrene Books.
21. Assylbekov, Z., Takhanov, R., Myrzakhmetov, B., and Washington, J.N. (2017, January 7–11). Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1199 ↗
22. Elwell, R. (2008, January 5). Using Syllables as Features in Morpheme Tagging in Swahili. Proceedings of the Fifth Midwest Computational Linguistics Colloquium, East Lansing, MI, USA.
23. A neural probabilistic language model
24. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA.
25. Vania, C., and Lopez, A. (August, January 30). From Characters to Words to in Between: Do We Capture Morphology?. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
26. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2016, January 1–5). Charagram: Embedding Words and Sentences via Character n-grams. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1157 ↗
27. Luong, T., Socher, R., and Manning, C. (2013, January 8–9). Better word representations with recursive neural networks for morphology. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, Sofia, Bulgaria.
28. Qiu, S., Cui, Q., Bian, J., Gao, B., and Liu, T.Y. (2014, January 23–29). Co-learning of word representations and morpheme representations. Proceedings of the 25th International Conference on Computational Linguistics, Dublin, Ireland.
29. Üstün, A., Kurfalı, M., and Can, B. (2018, January 20). Characters or Morphemes: How to Represent Words?. Proceedings of the Third Workshop on Representation Learning for NLP, Melbourne, Australia.10.18653/v1/W18-3019 ↗
30. Yu, S., Kulkarni, N., Lee, H., and Kim, J. (2017, January 7–11). Syllable-level Neural Language Model for Agglutinative Language. Proceedings of the First Workshop on Subword and Character Level Models in NLP, Copenhagen, Denmark.10.18653/v1/W17-4113 ↗
31. LeCun, Y., and Bengio, Y. (1995). Convolutional Networks for Images, Speech, and Time Series, The Handbook of Brain Theory and Neural Networks.
32. Srivastava, R.K., Greff, K., and Schmidhuber, J. (2015, January 6–11). Highway networks. Proceedings of the 2015 Deep Learning Workshop International Conference on Machine Learning, Lille, France.
33. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
34. Gelas, H., Besacier, L., and Pellegrino, F. (2012). Developments of Swahili Resources for An Automatic Speech Recognition System, SLTU—Workshop on Spoken Language Technologies for Under-Resourced Languages.
35. Polomé, E.C. (1967). Swahili Language Handbook, Center for Applied Linguistics.
36. De Pauw, G., and De Schryver, G.M. (2008). Improving the computational morphological analysis of a Swahili corpus for lexicographic purposes. Lexikos.10.4314/lex.v18i1.47257 ↗
37. Deen, K.U.D.S.U. (2002). The Acquisition of Nairobi Swahili: The Morphosyntax of Inflectional Prefixes and Subjects. [Ph.D. Thesis, University of California].
38. Semantic analysis of Kiswahili words using the self organizing Map
39. Kiswahili: People, language, literature and lingua franca
40. Masengo, I.J. (2018). Cross-Linguistic Influence in Third Language Production Among Kiswahili Learners. [Ph.D. Thesis, Makerere University].
41. Hurskainen, A. (1996, January 5–9). Disambiguation of morphological analysis in Bantu languages. Proceedings of the 16th Conference on Computational Linguistics-Volume 1, Copenhagen, Denmark.10.3115/992628.992726 ↗
42. De Pauw, G., De Schryver, G.M., and Wagacha, P.W. (2006, January 11–15). Data-driven part-of-speech tagging of Kiswahili. Proceedings of the International Conference on Text, Speech and Dialogue, Brno, Czech Republic.10.1007/11846406_25 ↗
43. Gas recognition under sensor drift by using deep learning10.1002/int.21731 ↗
44. Convolutional recurrent deep learning model for sentence classification10.1109/ACCESS.2018.2814818 ↗
45. Deep neural networks for acoustic modeling in speech recognition10.1109/MSP.2012.2205597 ↗
46. Finding structure in time10.1207/s15516709cog1402_1 ↗
47. Bidirectional recurrent neural networks10.1109/78.650093 ↗
48. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, The MIT Press.
49. Convolution-Based Neural Attention with Applications to Sentiment Classification10.1109/ACCESS.2019.2900335 ↗
50. Yih, W.T., He, X., and Meek, C. (2014, January 22–27). Semantic parsing for single-relation question answering. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, MD, USA.10.3115/v1/P14-2105 ↗
51. Shen, Y., He, X., Gao, J., Deng, L., and Mesnil, G. (2014, January 7–11). Learning semantic representations using convolutional neural networks for web search. Proceedings of the 23rd International Conference on World Wide Web, Seoul, Korea.10.1145/2567948.2577348 ↗
52. Santos, C.D., and Zadrozny, B. (2014, January 21–26). Learning character-level representations for part-of-speech tagging. Proceedings of the 31st International Conference on Machine Learning (ICML 2014), Beijing, China.
53. Xiao, Y., and Cho, K. (2016). Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers. arXiv.
54. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar.10.3115/v1/D14-1162 ↗
55. Cotterell, R., and Schütze, H. (June, January 31). Morphological word-embeddings. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, CO, USA.10.3115/v1/N15-1140 ↗
56. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018, January 7–12). Learning Word Vectors for 157 Languages. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan.
57. An extensive empirical evaluation of character-based morphological tagging for 14 languages
58. Zaremba, W., Sutskever, I., and Vinyals, O. (2014). Recurrent neural network regularization. arXiv.
59. Srivastava, R.K., Greff, K., and Schmidhuber, J. (2015). Training very deep networks. Advances in Neural Information Processing Systems, The MIT Press.
60. Bantu spirantization: Morphologization, lexicalization and historical classification