DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
Source abstract
Abstract
Word embedding is an important reference for natural language processing tasks, which can generate distribution presentations of words based on many text data. Recent evidence demonstrates that introducing sememe knowledge is a promising strategy to improve the performance of word embedding. However, previous works ignored the structure information of sememe knowledges. To fill the gap, this study implicitly synthesized the structural feature of sememes into word embedding models based on an attention mechanism. Specifically, we propose a novel double attention word-based embedding (DAWE) model that encodes the characteristics of sememes into words by a “double attention” strategy. DAWE is integrated with two specific word training models through context-aware semantic matching techniques. The experimental results show that, in word similarity task and word analogy reasoning task, the performance of word embedding can be effectively improved by synthesizing the structural information of sememe knowledge. The case study also verifies the power of DAWE model in word sense disambiguation task. Furthermore, the DAWE model is a general framework for encoding sememes into words, which can be integrated into other existing word embedding models to provide more options for various natural language processing downstream tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Geography and Information Engineering, China University of Geosciences, Wuhan 430074, China
National Engineering Research Center for Geographic Information System, University of Geosciences, Wuhan 430074, China
School of Computer Science, China University of Geosciences, Wuhan 430074, China
Funding
National Natural Science Foundation of China
References (30)
Show source-provided references
1. Bian, J., Gao, B., and Liu, T.-Y. (2014). Knowledge-Powered Deep Learning for Word Embedding, Springer.10.1007/978-3-662-44848-9_9 ↗
2. Cao, S., and Lu, W. (2017). Improving Word Embeddings with Convolutional Feature Learning and Subword Information, AAAI.10.1609/aaai.v31i1.10993 ↗
3. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013, January 12–15). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Sydney, Australia.
4. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the International Conference on Learning Representations, Scottsdale, Arizona.
5. Bilingual embeddings with random walks over multilingual wordnets10.1016/j.knosys.2018.03.017 ↗
6. Mancini, M., Camacho-Collados, J., Iacobacci, I., and Navigli, R. (2017, January 3–4). Embedding Words and Senses Together via Joint Knowledge-Enhanced Training. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, BC, Canada.10.18653/v1/K17-1012 ↗
7. Niu, Y., Xie, R., Liu, Z., and Sun, M. (August, January 30). Improved word representation learning with sememes. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
8. Tissier, J., Gravier, C., and Habrard, A. (2017, January 7–11). Dict2vec: Learning Word Embeddings using Lexical Dictionaries. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), Copenhagen, Denmark.10.18653/v1/D17-1024 ↗
9. Yang, L., and Sun, M. (2015). Improved learning of chinese word embeddings with semantic knowledge. Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, Springer.10.1007/978-3-319-25816-4_2 ↗
10. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (August, January 28). ERNIE: Enhanced Language Representation with Informative Entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
11. Dong, Z., and Dong, Q. (2003). Hownet-a hybrid language and knowledge resource. Proceedings of the International Conference on Natural Language Processing and Knowledge Engineering, Proceedings 2003, IEEE.
12. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, The MIT Press.
13. Chen, X., Xu, L., Liu, Z., Sun, M., and Luan, H. (2015, January 25–31). Joint learning of character and word embeddings. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
14. Sun, F., Guo, J., Lan, Y., Xu, J., and Cheng, X. (2016, January 12–17). Inside out: Two jointly predictive models for word representations and phrase representations. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10338 ↗
15. Xu, J., Liu, J., Zhang, L., Li, Z., and Chen, H. (2016, January 12–17). Improve chinese word embeddings by exploiting internal structure. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1119 ↗
16. Cao, S., and Lu, W. (2017, January 4–9). Improving word embeddings with convolutional feature learning and subword information. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10993 ↗
17. Li, S., Zhao, Z., Hu, R., Li, W., Liu, T., and Du, X. (2018, January 15–20). Analogical reasoning on chinese morphological and semantic relations. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-2023 ↗
18. Shi, X., Zhai, J., Yang, X., Xie, Z., and Liu, C. (2015, January 26–31). Radical embedding: Delving deeper to chinese radicals. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-2098 ↗
19. Yin, R., Wang, Q., Li, P., Li, R., and Wang, B. (2016, January 1–4). Multi-granularity chinese word embedding. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1100 ↗
20. Yu, J., Jian, X., Xin, H., and Song, Y. (2017, January 9–11). Joint embeddings of chinese words, characters, and fine-grained subcharacter components. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1027 ↗
21. Cao, S., Lu, W., Zhou, J., and Li, X. (2018, January 2–7). Cw2vec: Learning chinese word embeddings with stroke n-gram information. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12029 ↗
22. Wordnet: A lexical database for english10.1145/219717.219748 ↗
23. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network10.1016/j.artint.2012.07.001 ↗
24. Speer, R., Chin, J., and Havasi, C. (2017, January 4–9). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11164 ↗
25. Liu, F., Lu, H., Lo, C., and Neubig, G. (August, January 30). Learning character-level compositionality with visual features. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
26. Wang, S., Zhang, J., and Zong, C. (2018, January 2–7). Learning multimodal word representation via dynamic fusion methods. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12031 ↗
27. Mei, J., Zhu, Y., Gao, Y., and Yin, H. (1983). Tongyici Cilin (Dictionary of Synonymous Words), Shanghai Cishu Publishing.
28. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., and Wang, P. (2020). K-Bert: Enabling Language Representation with Knowledge graph, AAAI.10.1609/aaai.v34i03.5681 ↗
29. From word to sense embeddings: A survey on vector representations of meaning10.1613/jair.1.11259 ↗
30. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗