Visual and Phonological Feature Enhanced Siamese BERT for Chinese Spelling Error Correction
Source abstract
Abstract
Chinese Spelling Check (CSC) aims to detect and correct spelling errors in Chinese. Most CSC models rely on human-defined confusion sets to narrow the search space, failing to resolve errors outside the confusion set. However, most spelling errors in current benchmark datasets are character pairs in similar pronunciations. Errors in similar shapes and errors which are visually and phonologically irrelevant are not considered. Furthermore, widely-used automatically generated training data in CSC tasks leads to label leakage and unfair comparison between different methods. In this work, we propose a feature (visual and phonological) enhanced siamese BERT to (1) correct spelling errors without using confusion sets; (2) integrate phonological and visual features for CSC by a glyph graph; (3) improve performance for unseen spelling errors. To evaluate CSC methods fairly and comprehensively, we build a large-scale CSC dataset in which the number of samples in different error types is the same. The experimental results show that the proposed approach achieves better performance compared with previous state-of-the-art methods on three benchmark datasets and the new error-type balanced dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Handwritten Text Recognition Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Chemical Engineering and Light Industry, Guangdong University of Technology, Guangzhou 510006, China
School of Automation Engineering, University of Electronic Science and Technology of China, Xiyuan West Road 2006, Chengdu 611731, China
Department of Computer Science and Technology, Nanjing University, Nanjing 210023, China
Funding
National Natural Science Foundation of China
References (29)
Show source-provided references
1. Sun, L., Zong, C., Zhang, M., and Levow, G. (2014, January 20–21). Chinese Spelling Error Detection and Correction Based on Language Model, Pronunciation, and Shape. Proceedings of the Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, Wuhan, China.
2. Yu, L., Sui, Z., Zhang, Y., and Ng, V. (2015, January 30–31). HANSpeller++: A Unified Framework for Chinese Spelling Correction. Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2015, Beijing, China.
3. Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (November, January 31). A Hybrid Approach to Automatic Corpus Generation for Chinese Spelling Check. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
4. Semisupervised Feature Selection Based on Relevance and Redundancy Criteria10.1109/TNNLS.2016.2562670 ↗
5. Limited-energy output formation for multiagent systems with intermittent interactions10.1016/j.jfranklin.2021.06.009 ↗
6. Liao, D., Xu, J., Li, G., Huang, W., Liu, W., and Li, J. (February, January 27). Popularity Prediction on Online Articles with Deep Fusion of Temporal Process and Content Features. Proceedings of the The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, HI, USA.
7. Korhonen, A., Traum, D.R., and Màrquez, L. (August, January 28). Confusionset-guided Pointer Networks for Chinese Spelling Check. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy. Volume 1: Long Papers.
8. Cheng, X., Xu, W., Chen, K., Jiang, S., Wang, F., Wang, T., Chu, W., and Qi, Y. (2021, January 6). SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, online.10.18653/v1/2020.acl-main.81 ↗
9. Burstein, J., Doran, C., and Solorio, T. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA. Volume 1 (Long and Short Papers).
10. Yu, L., Lee, L., Tseng, Y., and Chen, H. (2014, January 20–21). Overview of SIGHAN 2014 Bake-off for Chinese Spelling Check. Proceedings of the Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, Wuhan, China.10.3115/v1/W14-6820 ↗
11. Hong, Y., Yu, X., He, N., Liu, N., and Liu, J. (2019, January 4). FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm. Proceedings of the 5th Workshop on Noisy User-Generated Text (W-NUT 2019), Hong Kong, China.10.18653/v1/D19-5522 ↗
12. Zhang, S., Huang, H., Liu, J., and Li, H. (2020, January 5–10). Spelling Error Correction with Soft-Masked BERT. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.82 ↗
13. Yu, L., Tseng, Y., Zhu, J., and Ren, F. (2013, January 14–18). A Hybrid Chinese Spelling Correction Using Language Model and Statistical Machine Translation with Reranking. Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2013, Nagoya, Japan.
14. Chen, H., Tseng, Y., Ng, V., and Lu, X. (2016, January 12). Chinese Grammatical Error Diagnosis with Long Short-Term Memory Networks. Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@COLING 2016, Osaka, Japan.
15. Liu, C., Nakov, P., and Xue, N. (December, January 27). Alibaba at IJCNLP-2017 Task 1: Embedding Grammatical Features into LSTMs for Chinese Grammatical Error Diagnosis Task. Proceedings of the IJCNLP 2017, Shared Tasks, Taipei, Taiwan.
16. Tseng, Y., Chen, H., Ng, V., and Komachi, M. (2018, January 19). CYUT-III Team Chinese Grammatical Error Diagnosis System Report in NLPTEA-2018 CGED Shared Task. Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL 2018, Melbourne, Australia.
17. Bao, Z., Li, C., and Wang, R. (2020). Chunk-based Chinese Spelling Check with Global Optimization. Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics.10.18653/v1/2020.findings-emnlp.184 ↗
18. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014, January 13). Empirical evaluation of gated recurrent neural networks on sequence modeling. Proceedings of the NIPS 2014 Workshop on Deep Learning, Montreal, QC, Canada.
19. Grover, A., and Leskovec, J. (2016, January 13–17). Node2vec: Scalable Feature Learning for Networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939754 ↗
20. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA.
21. Jiang, Z., Gao, Z., He, G., Kang, Y., Sun, C., Zhang, Q., Si, L., and Liu, X. (2019, January 3–7). Detect Camouflaged Spam Content via StoneSkipping: Graph and Text Joint Embedding for Chinese Character Variation Representation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1640 ↗
22. Zong, C., Xia, F., Li, W., and Navigli, R. (2021). Correcting Chinese Spelling Errors with Phonetic Pre-training. Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, 1–6 August 2021, Association for Computational Linguistics. Volume ACL/IJCNLP 2021.
23. Wu, S.H., Liu, C.L., and Lee, L.H. (2013, January 14–18). Chinese Spelling Check Evaluation at SIGHAN Bake-off 2013. Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, Nagoya, Japan.
24. Tseng, Y.H., Lee, L.H., Chang, L.P., and Chen, H.H. (2015, January 30–31). Introduction to SIGHAN 2015 Bake-off for Chinese Spelling Check. Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, Beijing, China.10.18653/v1/W15-3106 ↗
25. Gulcehre, C., Ahn, S., Nallapati, R., Zhou, B., and Bengio, Y. (2016, January 7–12). Pointing the Unknown Words. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1014 ↗
26. Loshchilov, I., and Hutter, F. (2019, January 6–9). Decoupled Weight Decay Regularization. Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. OpenReview.net, 2019.
27. Silva, R., Globerson, A., and Globerson, A. (2018, January 6–10). Averaging weights leads to wider optima and better generalization. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, CA, USA.
28. Xie, W., Huang, P., Zhang, X., Hong, K., Huang, Q., Chen, B., and Huang, L. (2015, January 30–31). Chinese Spelling Check System Based on N-gram Model. Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, SIGHAN@IJCNLP 2015, Beijing, China.10.18653/v1/W15-3120 ↗
29. Bertini, E., Elmqvist, N., and Wischgoll, T. (2016, January 6–10). Visualizing Time-Dependent Data Using Dynamic t-SNE. Proceedings of the 18th Eurographics Conference on Visualization, EuroVis 2016—Short Papers, Groningen, The Netherlands.