A Multi-Granularity Word Fusion Method for Chinese NER
Source abstract
Abstract
Named entity recognition (NER) plays a crucial role in many downstream natural language processing (NLP) tasks. It is challenging for Chinese NER because of certain features of Chinese. Recently, large-scaled pre-training language models have been used in Chinese NER. However, since some of the pre-training language models do not use word information or just employ word information of single granularity, the semantic information in sentences could not be fully captured, which affects these models’ performance. To fully take advantage of word information and obtain richer semantic information, we propose a multi-granularity word fusion method for Chinese NER. We introduce multi-granularity word information into our model. To make full use of the information, we classify the information into three kinds: strong information, moderate information, and weak information. These kinds of information are encoded by encoders and then integrated with each other through the strong-weak feedback attention mechanism. Specifically, we apply two separate attention networks to word embeddings and N-grams embeddings. Then, the outputs are fused into another attention. In these three attentions, character embeddings are used to be the query of attentions. We call the results the multi-granularity word information. To combine character information and multi-granularity word information, we introduce two fusion strategies for better performance. The process makes our model obtain rich semantic information and reduces word segmentation errors and noise in an explicit way. We design experiments to get our model’s best performance by comparing some components. Ablation study is used to verify the effectiveness of each module. The final experiments are conducted on four Chinese NER benchmark datasets and the F1 scores are 81.51% for Ontonotes4.0, 95.47% for MSRA, 95.87% for Resume, and 69.41% for Weibo. The best improvement achieved by the proposed method is 1.37%. Experimental results show that our method outperforms most baselines and achieves the state-of-the-art method in performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
Shandong Key Laboratory of Wisdom Mine Information Technology, Qingdao 266590, China
Funding
National Natural Science Foundation of China · Shandong Provincial Natural Science Foundation · Qingdao Philosophy and Social Sciences Planning Project · Talented Young Teachers Training Program of Shandong University of Science and Technology
References (32)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
2. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
3. Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., and Wu, H. (2019). ERNIE: Enhanced Representation through Knowledge Integration. arXiv.
4. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗
5. Diao, S., Bai, J., Song, Y., Zhang, T., and Wang, Y. (2020, January 16–20). ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA.10.18653/v1/2020.findings-emnlp.425 ↗
6. Tian, Y., Song, Y., Xia, F., Zhang, T., and Wang, Y. (2020, January 5–10). Improving Chinese Word Segmentation with Wordhood Memory Networks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.734 ↗
7. Natural language processing (almost) from scratch
8. Santos, C.N.d., and Guimaraes, V. (2015). Boosting named entity recognition with neural character embeddings. arXiv.10.18653/v1/W15-3904 ↗
9. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016, January 12–17). Neural Architectures for Named Entity Recognition. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1030 ↗
10. Ma, X., and Hovy, E. (2016, January 7–12). End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1101 ↗
11. Named Entity Recognition with Bidirectional LSTM-CNNs10.1162/tacl_a_00104 ↗
12. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
13. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
14. Zhu, W., and Cheung, D. (2021). Lex-BERT: Enhancing BERT based NER with lexicons. arXiv.
15. Liu, W., Fu, X., Zhang, Y., and Xiao, W. (2021, January 1–6). Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.10.18653/v1/2021.acl-long.454 ↗
16. Zhang, Y., and Yang, J. (2018, January 15–20). Chinese NER Using Lattice LSTM. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1144 ↗
17. Li, X., Yan, H., Qiu, X., and Huang, X. (2020, January 5–10). FLAT: Chinese NER Using Flat-Lattice Transformer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.611 ↗
18. Song, C., Xiong, Y., Huang, W., and Ma, L. (2020, January 24–25). Joint Self-Attention and Multi-Embeddings for Chinese Named Entity Recognition. Proceedings of the 2020 6th International Conference on Big Data Computing and Communications (BIGCOM), Deqing, China.10.1109/BigCom51056.2020.00017 ↗
19. Johnson, S., Shen, S., and Liu, Y. (2020). CWPC_BiAtt: Character–Word–Position Combined BiLSTM-Attention for Chinese Named Entity Recognition. Information, 11.10.3390/info11010045 ↗
20. Mengge, X., Yu, B., Liu, T., Zhang, Y., Meng, E., and Wang, B. (2020, January 8–13). Porous Lattice Transformer Encoder for Chinese NER. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.340 ↗
21. Wu, S., Song, X., and Feng, Z. (2021, January 1–6). MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1): Long Papers, Online.10.18653/v1/2021.acl-long.121 ↗
22. Yu, J., and Jiang, J. (2019, January 10–16). Adapting BERT for Target-Oriented Multimodal Sentiment Classification. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/751 ↗
23. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5–10). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
24. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
25. Su, J. (2022, April 17). WoBERT: Word-Based Chinese BERT Model—ZhuiyiAI. Technical Report. Available online: https://zhuiyi.ai/.
26. Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., and Franchini, M. (2011). OntoNotes Release 4.0, Linguistic Data Consortium.
27. Levow, G.A. (2006, January 22–23). The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition. Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, Sydney, Australia.
28. Peng, N., and Dredze, M. (2015, January 17–21). Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1064 ↗
29. He, H., and Sun, X. (2017, January 3–7). F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Media. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain.10.18653/v1/E17-2113 ↗
30. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
31. Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional LSTM-CRF models for sequence tagging. arXiv.
32. Geng, Z., Yan, H., Qiu, X., and Huang, X. (2020). fastHan: A BERT-based Multi-Task Toolkit for Chinese NLP. arXiv.10.18653/v1/2021.acl-demo.12 ↗