journal article·2022-07-30Open access

KGNER: Improving Chinese Named Entity Recognition by BERT Infused with the Knowledge Graph

Weiwei Hu, Liang He, Hanhan Ma, Kai Wang, Jingfeng Xiao

Applied Sciences· Volume 12· Issue 15

Abstract

Recently, the lexicon method has been proven to be effective for named entity recognition (NER). However, most existing lexicon-based methods cannot fully utilize common-sense knowledge in the knowledge graph. For example, the word embeddings pretrained by Word2vector or Glove lack better contextual semantic information usage. Hence, how to make the best of knowledge for the NER task has become a challenging and hot research topic. We propose a knowledge graph-inspired named-entity recognition (KGNER) featuring a masking and encoding method to incorporate common sense into bidirectional encoder representations from transformers (BERT). The proposed method not only preserves the original sentence semantic information but also takes advantage of the knowledge information in a more reasonable way. Subsequently, we model the temporal dependencies by taking the conditional random field (CRF) as the backend, and improve the overall performance. Experiments on four dominant datasets demonstrate that the KGNER outperforms other lexicon-based models in terms of performance.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Data Quality and Management

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

College of Information Science and Engineering, Xinjiang University, Urumqi 830046, China

Department of Electronic Engineering, Tsinghua University, Beijing 100084, China

State Grid Xinjiang Electric Power Co., Ltd., Urumqi 830000, China

References (56)

Show source-provided references
1. Chen, Y., Xu, L., Liu, K., Zeng, D., and Zhao, J. (2015, January 26–31). Event extraction via dynamic multi-pooling convolutional neural networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1017 ↗
2. Mao, N., Huang, W., and Zhong, H. (2021). KGGCN: Knowledge-Guided Graph Convolutional Networks for Distantly Supervised Relation Extraction. Appl. Sci., 11.10.3390/app11167734 ↗
3. Kim, K., Hur, Y., Kim, G., and Lim, H. (2020). GREG: A Global Level Relation Extraction with Knowledge Graph Embedding. Appl. Sci., 10.10.3390/app10031181 ↗
4. Core techniques of question answering systems over knowledge bases: A survey10.1007/s10115-017-1100-y ↗
5. He, J., and Wang, H. (2008, January 11–12). Chinese named entity recognition and word segmentation based on character. Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing, Hyderabad, India.
6. Liu, Z., Zhu, C., and Zhao, T. (2010, January 22–23). Chinese named entity recognition with a sequence labeling approach: Based on characters, or based on words?. Proceedings of the International Conference on Intelligent Computing, Washington, DC, USA.10.1007/978-3-642-14932-0_78 ↗
7. Luo, G., Huang, X., Lin, C.Y., and Nie, Z. (2015, January 17–21). Joint entity recognition and disambiguation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1104 ↗
8. Zhang, Y., and Yang, J. (2018). Chinese NER using lattice LSTM. arXiv.10.18653/v1/P18-1144 ↗
9. Gui, T., Ma, R., Zhang, Q., Zhao, L., Jiang, Y.G., and Huang, X. (2019, January 10–16). CNN-Based Chinese NER with Lexicon Rethinking. Proceedings of the IJCAI, Macao, China.10.24963/ijcai.2019/692 ↗
10. Gui, T., Zou, Y., Zhang, Q., Peng, M., Fu, J., Wei, Z., and Huang, X. (2019, January 3–7). A Lexicon-Based Graph Neural Network for Chinese NER. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1096 ↗
11. He, Q., Wu, L., Yin, Y., and Cai, H. (2020, January 7–12). Knowledge-graph augmented word representations for named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6299 ↗
12. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., and Wang, P. (2020, January 7–12). K-bert: Enabling language representation with knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i03.5681 ↗
13. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
14. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
15. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019). ERNIE: Enhanced language representation with informative entities. arXiv.10.18653/v1/P19-1139 ↗
16. Huang, S., Sun, X., and Wang, H. (December, January 27). Addressing domain adaptation for chinese word segmentation with global recurrent structure. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan.
17. Chen, X., Shi, Z., Qiu, X., and Huang, X. (2017). Adversarial multi-criteria learning for chinese word segmentation. arXiv.10.18653/v1/P17-1110 ↗
18. Ma, R., Peng, M., Zhang, Q., and Huang, X. (2019). Simplify the usage of lexicon in Chinese NER. arXiv.10.18653/v1/2020.acl-main.528 ↗
19. Mengge, X., Bowen, Y., Tingwen, L., Yue, Z., Erli, M., and Bin, W. (2019). Porous lattice-based transformer encoder for chinese NER. arXiv.10.18653/v1/2020.coling-main.340 ↗
20. Sui, D., Chen, Y., Liu, K., Zhao, J., and Liu, S. (2019, January 3–7). Leverage lexical knowledge for Chinese named entity recognition via collaborative graph network. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1396 ↗
21. Li, X., Yan, H., Qiu, X., and Huang, X. (2020). FLAT: Chinese NER using flat-lattice transformer. arXiv.10.18653/v1/2020.acl-main.611 ↗
22. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
23. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
24. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2022, July 22). Improving Language Understanding by Generative Pre-training. Available online: https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035.
25. Yang, H. (2019). BERT meets chinese word segmentation. arXiv.
26. Meng, Y., Wu, W., Wang, F., Li, X., Nie, P., Yin, F., Li, M., Han, Q., Sun, X., and Li, J. (2019). Glyce: Glyph-vectors for chinese character representations. arXiv.
27. Hu, Y., Verberne, S., Scott, D., Bel, N., and Zong, C. (2020, January 8–13). Named entity recognition for Chinese biomedical patents. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.54 ↗
28. Liu, W., Fu, X., Zhang, Y., and Xiao, W. (2021). Lexicon enhanced chinese sequence labeling using bert adapter. arXiv.10.18653/v1/2021.acl-long.454 ↗
29. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
30. Exploring the limits of transfer learning with a unified text-to-text transformer
31. Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. (2022, January 22–27). GLM: General Language Model Pretraining with Autoregressive Blank Infilling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.26 ↗
32. Yan, H., Gui, T., Dai, J., Guo, Q., Zhang, Z., and Qiu, X. (2021). A unified generative framework for various NER subtasks. arXiv.10.18653/v1/2021.acl-long.451 ↗
33. Cui, L., Wu, Y., Liu, J., Yang, S., and Zhang, Y. (2021). Template-based named entity recognition using BART. arXiv.10.18653/v1/2021.findings-acl.161 ↗
34. Xu, B., Xu, Y., Liang, J., Xie, C., Liang, B., Cui, W., and Xiao, Y. (2017, January 27–30). CN-DBpedia: A never-ending Chinese knowledge extraction system. Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Arras, France.10.1007/978-3-319-60045-1_44 ↗
35. Dong, Z., and Dong, Q. (2003, January 26–29). HowNet-a hybrid language and knowledge resource. Proceedings of the International Conference on Natural Language Processing and Knowledge Engineering, Beijing, China.
36. Bordes, A., Usunier, N., Garcia-Durán, A., Weston, J., and Yakhnenko, O. (2013, January 5–10). Translating Embeddings for Modeling Multi-Relational Data. Proceedings of the 26th International Conference on Neural Information Processing Systems—Volume 2, Lake Tahoe, NV, USA.
37. Xie, R., Liu, Z., Jia, J., Luan, H., and Sun, M. (2016, January 12–17). Representation learning of knowledge graphs with entity descriptions. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10329 ↗
38. Zhong, H., Zhang, J., Wang, Z., Wan, H., and Chen, Z. (2015, January 17–21). Aligning knowledge and text embeddings by entity descriptions. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1031 ↗
39. Xiao, H., Huang, M., Hao, Y., and Zhu, X. (2015). Transg: A generative mixture model for knowledge graph embedding. arXiv.10.18653/v1/P16-1219 ↗
40. Ji, G., He, S., Xu, L., Liu, K., and Zhao, J. (2015, January 26–31). Knowledge graph embedding via dynamic mapping matrix. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1067 ↗
41. Shi, B., and Weninger, T. (2017, January 4–9). Proje: Embedding projection for knowledge graph completion. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10677 ↗
42. Annervaz, K., Chowdhury, S.B.R., and Dukkipati, A. (2018). Learning beyond datasets: Knowledge graph augmented neural networks for natural language processing. arXiv.
43. Xin, J., Lin, Y., Liu, Z., and Sun, M. (2018, January 2–7). Improving neural fine-grained entity typing with knowledge attention. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12038 ↗
44. Wang, Y., Sun, Y., Ma, Z., Gao, L., and Xu, Y. (2020). An ERNIE-Based Joint Model for Chinese Named Entity Recognition. Appl. Sci., 10.10.3390/app10165711 ↗
45. Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Cao, G., Jiang, D., and Zhou, M. (2020). K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv.10.18653/v1/2021.findings-acl.121 ↗
46. Sang, E.F., and Veenstra, J. (1999). Representing text chunks. arXiv.10.3115/977035.977059 ↗
47. Khan, W., Daud, A., Shahzad, K., Amjad, T., Banjar, A., and Fasihuddin, H. (2022). Named Entity Recognition Using Conditional Random Fields. Appl. Sci., 12.10.3390/app12136391 ↗
48. Lafferty, J.D., McCallum, A., and Pereira, F.C.N. (July, January 28). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning, ICML `01, San Diego, CA, USA.
49. Weischedel, R., Pradhan, S., Ramshaw, L., Palmer, M., Xue, N., Marcus, M., Taylor, A., Greenberg, C., Hovy, E., and Belvin, R. (2011). Ontonotes Release 4.0, Linguistic Data Consortium. LDC2011T03.
50. Levow, G.A. (2006, January 22–23). The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, Sydney, Australia.
51. Peng, N., and Dredze, M. (2015, January 17–21). Named entity recognition for chinese social media with jointly trained embeddings. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1064 ↗
52. He, H., and Sun, X. (2016). F-score driven max margin neural network for named entity recognition in chinese social media. arXiv.10.18653/v1/E17-2113 ↗
53. Zhu, Y., Wang, G., and Karlsson, B.F. (2019). CAN-NER: Convolutional attention network for Chinese named entity recognition. arXiv.
54. Liu, W., Xu, T., Xu, Q., Song, J., and Zu, Y. (June, January 2). An encoding strategy based word-character LSTM for Chinese NER. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
55. Ding, R., Xie, P., Zhang, X., Lu, W., Li, L., and Si, L. (August, January 28). A neural multi-digraph model for Chinese NER with gazetteers. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
56. Diao, S., Bai, J., Song, Y., Zhang, T., and Wang, Y. (2019). ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. arXiv.10.18653/v1/2020.findings-emnlp.425 ↗