Word Vector Representation of Latin Cuengh Based on Root Feature Enhancement
Source abstract
Abstract
The Latin Cuengh is a kind of language used in China’s minority areas. Due to its complex pronunciation and semantic system, it is difficult to spread widely. To deal with and protect this language further, this paper considers using the current word vector representation technology to study it. Word vector representation is the basic method and an important foundation of current research on natural language processing. It relies on a large number of data resources and is obtained through the paradigm of pre-training and feature learning. Due to the extreme lack of Latin Cuengh corpus resources, it is very difficult to obtain word vectors by relying on a large amount of data training. In this study, we propose a word vector representation method that combines the root features of Latin Cuengh words. Specifically, while training and learning the Latin Cuengh language corpus, this method uses the special word roots in the Latin Cuengh language to modify the training process, which can enhance the expression ability of the root features. The method uses the mask method based on BERT to mask the word roots after word segmentation and predict the masked word roots in the output layer of the model to obtain a better vector representation of Latin Cuengh words. The experimental results show that the word vector representation method proposed in this paper is effective and has the ability to express Latin Cuengh semantics. The accuracy rate of words semantic is nearly 2% points higher than that of BERT representation, and the judgment of the semantic similarity of words is more accurate.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology, Guilin 541004, China
Funding
Guangxi Natural Science Foundation · Guangxi Key Research and Development Program · Guangxi Key Laboratory of Image and Graphic Intelligent Processing · National Natural Science Foundation of China
References (40)
Show source-provided references
1. Learning word representation by jointly using neighbor and syntactic contexts10.1016/j.neucom.2021.03.130 ↗
2. Learning Chinese word representation better by cascade morphological n-gram10.1007/s00521-020-05198-7 ↗
3. A brief overview of universal sentence representation methods: A linguistic view
4. Semantic change detection with gaussian word embeddings10.1109/TASLP.2021.3120645 ↗
5. Adaptive cross-contextual word embedding for word polysemy with unsupervised topic modeling10.1016/j.knosys.2021.106827 ↗
6. TransPhrase: A new method for generating phrase embedding from word embedding in Chinese10.1016/j.eswa.2020.114387 ↗
7. Sun, L. (2021, January 7–9). Unsupervised Feature Representation Learning using Sequence-to-sequence Autoencoder Architecture for Low-resource Language. Proceedings of the 2021 International Conference on Computer Communication and Artificial Intelligence (CCAI), Guangzhou, China.10.1109/CCAI50917.2021.9447504 ↗
8. Tiyajamorn, N., Kajiwara, T., Arase, Y., and Onizuka, M. (2021, January 7–11). Language-agnostic representation from multilingual sentence encoders for cross-lingual similarity estimation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.612 ↗
9. Doumbouya, M., Einstein, L., and Piech, C. (2021, January 19–21). Using radio archives for low-resource speech recognition: Towards an intelligent virtual assistant for illiterate users. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i17.17733 ↗
10. Canine: Pre-training an efficient tokenization-free encoder for language representation10.1162/tacl_a_00448 ↗
11. Fakharian, S., and Cook, P. (2021, January 6). Contextualized embeddings encode monolingual and cross-lingual knowledge of idiomaticity. Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), Bangkok, Thailand.10.18653/v1/2021.mwe-1.4 ↗
12. Shwartz, V. (2021, January 6). A long hard look at MWEs in the age of language models. Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), Bangkok, Thailand.10.18653/v1/2021.mwe-1.1 ↗
13. Kranzlein, M., Liu, N.F., and Schneider, N. (2021). Making heads and tails of models with marginal calibration for sparse tagsets. arXiv.10.18653/v1/2021.findings-emnlp.423 ↗
14. Li, I., Sen, P., Zhu, H., Li, Y., and Radev, D. (2021, January 6). Improving cross-lingual text classification with zero-shot instance-weighting. Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), Online.10.18653/v1/2021.repl4nlp-1.1 ↗
15. Goyal, N., Du, J., Ott, M., Anantharaman, G., and Conneau, A. (2021). Larger-scale transformers for multilingual masked language modeling. arXiv.10.18653/v1/2021.repl4nlp-1.4 ↗
16. Bałazy, K., Banaei, M., Lebret, R., Tabor, J., and Aberer, K. (2021). Direction is what you need: Improving word embedding compression in large language models. arXiv.10.18653/v1/2021.repl4nlp-1.32 ↗
17. Winata, G.I., Madotto, A., Lin, Z., Liu, R., Yosinski, J., and Fung, P. (2021). Language models are few-shot multilingual learners. arXiv.10.18653/v1/2021.mrl-1.1 ↗
18. Jegadeesan, M., Kumar, S., Wieting, J., and Tsvetkov, Y. (2021). Improving the diversity of unsupervised paraphrasing with embedding outputs. arXiv.10.18653/v1/2021.mrl-1.15 ↗
19. Prasad, A., Rehan, M.A., Pathak, S., and Jyothi, P. (2021). The effectiveness of intermediate-task training for code-switched natural language understanding. arXiv.10.18653/v1/2021.mrl-1.16 ↗
20. Embeddings in natural language processing: Theory and advances in vector representations of meaning10.1162/coli_r_00410 ↗
21. Natural language processing and computational linguistics
22. Lifeng, L. (2020). A Study on Reference Grammar of Guangxi Cuengh Language. [Ph.D. Thesis, Shanghai Normal University].
23. Aiyun, W. (2019). A Metrological Study of Cuengh Language Vocabulary System. [Ph.D. Thesis, Zhejiang University].
24. Aoben, X. (2020). The Mataphor Pattern and Path of Visual Word in Cuengh Language: Taking Cuengh Language in Mashan as an Example. Guangxi Ethn. Stud., 150–156.
25. Mikolov, T. (2013). Efficient estimation of word representations in vector space. arXiv.
26. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst., 26.
27. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
28. Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
29. Ogueji, K., Zhu, Y., and Lin, J. (2021, January 11). Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. Proceedings of the 1st Workshop on Multilingual Representation Learning, Punta Cana, Dominican Republic.10.18653/v1/2021.mrl-1.11 ↗
30. Zhang, Y., Warstadt, A., Li, H.S., and Bowman, S.R. (2020). When do you need billions of words of pretraining data?. arXiv.10.18653/v1/2021.acl-long.90 ↗
31. Wada, T., Iwata, T., Matsumoto, Y., Baldwin, T., and Lau, J.H. (2020). Learning contextualised cross-lingual word embeddings and alignments for extremely low-resource languages using parallel corpora. arXiv.10.18653/v1/2021.mrl-1.2 ↗
32. Cortis, K., Freitas, A., Daudert, T., Huerlimann, M., Zarrouk, M., Handschuh, S., and Davis, B. (2017, January 3–4). Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, BC, Canada.10.18653/v1/S17-2089 ↗
33. Yang, Y., Uy, M.C.S., and Huang, A. (2020). Finbert: A pretrained language model for financial communications. arXiv.
34. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., and Wang, P. (2020, January 7–12). K-bert: Enabling language representation with knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
35. Knowledge-enabled BERT for aspect-based sentiment analysis10.1016/j.knosys.2021.107220 ↗
36. Sahrawat, D., Mahata, D., Zhang, H., Kulkarni, M., Sharma, A., Gosangi, R., Stent, A., Kumar, Y., Shah, R.R., and Zimmermann, R. (2020). Keyphrase Extraction as Sequence Labeling Using Contextualized Embeddings, Springer.10.1007/978-3-030-45442-5_41 ↗
37. Liang, X., Wu, S., Li, M., and Li, Z. (2021). Unsupervised keyphrase extraction by jointly modeling local and global context. arXiv.10.18653/v1/2021.emnlp-main.14 ↗
38. Complex network based supervised keyword extractor10.1016/j.eswa.2019.112876 ↗
39. Gero, Z., and Ho, J.C. (2021, January 6–11). Word centrality constrained representation for keyphrase extraction. Proceedings of the Conference. Association for Computational Linguistics. North American Chapter. Meeting. NIH Public Access, Online.10.18653/v1/2021.bionlp-1.17 ↗
40. Shen, X., Wang, Y., Meng, R., and Shang, J. (March, January 22). Unsupervised Deep Keyphrase Generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.