RSVN: A RoBERTa Sentence Vector Normalization Scheme for Short Texts to Extract Semantic Information
Source abstract
Abstract
With the explosive growth in short texts on the Web and an increasing number of Web corpora consisting of short texts, short texts are playing an important role in various Web applications. Entity linking is a crucial task in knowledge graphs and a key technology in the field of short texts that affects the accuracy of many downstream tasks in natural language processing. However, compared to long texts, the entity-linking task of Chinese short text is a challenging problem due to the serious colloquialism and insufficient contexts. Moreover, existing methods for entity linking in Chinese short text underutilize semantic information and ignore the interaction between label information and the original short text. In this paper, we propose a RoBERTa sentence vector normalization scheme for short texts to fully extract the semantic information. Firstly, the proposed model utilizes RoBERTa to fully capture contextual semantic information. Secondly, the anisotropy of RoBERTa’s output sentence vectors is revised by utilizing the standard Gaussian of flow model, which enables the sentence vectors to more precisely characterize the semantics. In addition, the interaction between label embedding and text embedding is employed to improve the NIL entity classification. Experimental results demonstrate that the proposed model outperforms existing research results and mainstream deep learning methods for entity linking in two Chinese short text datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Graph Neural Networks · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China
School of Biological and Chemical Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China
Funding
Soft Science Research Program of Zhejiang Province · Zhejiang Province Key Research and Development Project
References (51)
Show source-provided references
1. Jiang, H., Gurajada, S., Lu, Q., Neelam, S., Popa, L., Sen, P., Li, Y., and Gray, A. (2021). LNN-EL: A Neuro-Symbolic Approach to Short-text Entity Linking. arXiv.10.18653/v1/2021.acl-long.64 ↗
2. Gu, Y., Qu, X., Wang, Z., Huai, B., Yuan, N.J., and Gui, X. (2021). Read, retrospect, select: An MRC framework to short text entity linking. arXiv.10.1609/aaai.v35i14.17528 ↗
3. Gupta, N., Singh, S., and Roth, D. (2017, January 7–11). Entity linking via joint encoding of types, descriptions, and context. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1284 ↗
4. Gillick, D., Kulkarni, S., Lansing, L., Presta, A., Baldridge, J., Ie, E., and Garcia-Olano, D. (2019). Learning dense representations for entity retrieval. arXiv.10.18653/v1/K19-1049 ↗
5. Logeswaran, L., Chang, M.-W., Lee, K., Toutanova, K., Devlin, J., and Lee, H. (2019). Zero-shot entity linking by reading entity descriptions. arXiv.10.18653/v1/P19-1335 ↗
6. Ou, J., Liu, N.N., Kai, Z., Yu, Y., and Yang, Q. (2011, January 24–28). Transferring Topical Knowledge from Auxiliary Long Texts for Short Text Clustering. Proceedings of the 20th ACM Conference on Information & Knowledge Management, Glasgow, Scotland, UK.
7. Bunescu, R., and Pasca, M. (2006, January 3–7). Using encyclopedic knowledge for named entity disambiguation. Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL-06), Trento, Italy.
8. Mann, G., and Yarowsky, D. (June, January 27). Unsupervised personal name disambiguation. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, Edmonton, AB, Cannada.10.3115/1119176.1119181 ↗
9. Zhang, W., Tan, C.L., Sim, Y.C., and Su, J. (2010, January 15–16). NUS-I2R: Learning a Combined System for Entity Linking. Proceedings of the 3th Text Analysis Conference, Gaithersburg, MD, USA.
10. Yupeng, J., Hongxu, H., and Ping, Y. (2015, January 26–28). LSA-Based Chinese-Slavic Mongolian NER Disambiguation. Proceedings of the 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, Liverpool, UK.10.1109/CIT/IUCC/DASC/PICOM.2015.102 ↗
11. Indexing by Latent Semantic Analysis10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 ↗
12. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
13. A novel method using word vector and graphical models for entity disambiguation in specific topic domains
14. Ganea, O.-E., and Hofmann, T. (2017). Deep joint entity disambiguation with local neural attention. arXiv.10.18653/v1/D17-1277 ↗
15. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
16. McCann, B., Bradbury, J., Xiong, C., and Socher, R. (2017). Learned in Translation: Contextualized Word Vectors. Adv. Neural Inf. Process. Syst., 30.
17. A Greedy Deep Learning Method for Medical Disease Analysis10.1109/ACCESS.2018.2823979 ↗
18. He, Z., Liu, S., Li, M., Zhou, M., Zhang, L., and Wang, H. (2013, January 4–9). Learning entity representation for entity disambiguation. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria.
19. Cao, Y., Hou, L., Li, J., and Liu, Z. (2018). Neural collective entity linking. arXiv.
20. CoGCN: Combining co-attention with graph convolutional network for entity linking with knowledge graphs10.1111/exsy.12606 ↗
21. Wu, J., Zhang, R., Mao, Y., Guo, H., Soflaei, M., and Huai, J. (2020, January 20–24). Dynamic graph convolutional networks for entity linking. Proceedings of the Web Conference 2020, Taipei, China.10.1145/3366423.3380192 ↗
22. Multi-Step Data Prediction in Wireless Sensor Networks Based on One-Dimensional CNN and Bidirectional LSTM10.1109/ACCESS.2019.2937098 ↗
23. Entity linking on Chinese microblogs via deep neural network10.1109/ACCESS.2018.2833153 ↗
24. Entity linking via symmetrical attention-based neural network and entity structural features
25. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
26. Fine-grained entity typing for domain independent entity linking
27. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
28. Improving entity linking by modeling latent entity type information
29. Cheng, J., Pan, C., Dang, J., Yang, Z., Guo, X., Zhang, L., and Zhang, F. (2019, January 24–27). Entity linking for Chinese short texts based on BERT and entity name embeddings. Proceedings of the 2019 China Conference on Knowledge Graph and Semantic Computing (CCKS2019), Hangzhou, China.
30. Entity Linking Via BERT and TextRank Keyword Extraction
31. Zhao, Y., Wang, Y., and Yang, N. (2020, January 9–11). Chinese Short Text Entity Linking Based On Semantic Similarity and Entity Correlation. Proceedings of the 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), Baltimore, MD, USA.10.1109/ICTAI50040.2020.00073 ↗
32. Chinese Short Text Entity Disambiguation Based on the Dual-Channel Hybrid Network10.1109/ACCESS.2020.3037333 ↗
33. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized bert pretraining approach. arXiv.
34. Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.10.18653/v1/D19-1410 ↗
35. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. arXiv.10.18653/v1/2020.emnlp-main.733 ↗
36. Xiong, Y., Feng, Y., Wu, H., Kamigaito, H., and Okimura, M. (2021, January 1–6). Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Bangkok, Thailand.10.18653/v1/2021.findings-acl.152 ↗
37. Phan, M.C., Sun, A., Tay, Y., Han, J., and Li, C. (2017, January 6–10). NeuPL: Attention-based semantic matching and pair-linking for entity disambiguation. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore.10.1145/3132847.3132963 ↗
38. Entity extraction, linking, classification, and tagging for social media: A wikipedia-based approach10.14778/2536222.2536237 ↗
39. Urata, T., and Maeda, A. (2017, January 9–13). An entity disambiguation approach based on wikipedia for entity linking in microblogs. Proceedings of the 2017 6th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), Hamamatsu, Japan.10.1109/IIAI-AAI.2017.171 ↗
40. Nozza, D., Sas, C., Fersini, E., and Messina, E. (2019, January 28–30). Word embeddings for unsupervised named entity linking. Proceedings of the 12th International Conference on Knowledge Science, Engineering and Management, Athens, Greece.10.1007/978-3-030-29563-9_13 ↗
41. Tan, C., Wei, F., Ren, P., Lv, W., and Zhou, M. (2017). Entity linking for queries by searching wikipedia sentences. arXiv.10.18653/v1/D17-1007 ↗
42. Munnelly, G., and Lawless, S. (2018, January 3–7). Investigating entity linking in early english legal documents. Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, Fort Worth, TX, USA.10.1145/3197026.3197055 ↗
43. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., and Polpsukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Congerence on Neural Information Processing Systems, Long Brach, CA, USA.
44. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗
45. Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.Y. (2019). Representation degeneration problem in training natural language generation models. arXiv.
46. Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., and Gu, Q. (2019, January 6–9). Improving neural language generation with spectrum control. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
47. Ethayarajh, K. (2019). How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv.10.18653/v1/D19-1006 ↗
48. Dinh, L., Krueger, D., and Bengio, Y. (2014). NICE: Non-linear Independent Components Estimation. arXiv.
49. Deep semantic match model for entity linking using knowledge graph and text10.1016/j.procs.2018.03.057 ↗
50. Lu, W., Zhou, Y., Lu, H., Ma, P., Zhang, Z., and Wei, B. (2017, January 8–12). Boosting collective entity linking via type-guided semantic embedding. Proceedings of the National CCF Conference on Natural Language Processing and Chinese Computing, Dalian, China.10.1007/978-3-319-73618-1_45 ↗
51. Entity Linking Based on Sentence Representation10.1155/2021/8895742 ↗