A Chinese Grammatical Error Correction Method Based on Iterative Training and Sequence Tagging
Source abstract
Abstract
Chinese grammatical error correction (GEC) is under continuous development and improvement, and this is a challenging task in the field of natural language processing due to the high complexity and flexibility of Chinese grammar. Nowadays, the iterative sequence tagging approach is widely applied to Chinese GEC tasks because it has a faster inference speed than sequence generation approaches. However, the training phase of the iterative sequence tagging approach uses sentences for only one round, while the inference phase is an iterative process. This makes the model focus only on the current sentence’s current error correction results rather than considering the results after multiple rounds of correction. In order to address this problem of mismatch between the training and inference processes, we propose a Chinese GEC method based on iterative training and sequence tagging (CGEC-IT). First, in the iterative training phase, we dynamically generate the target tags for each round by using the final target sentences and the input sentences of the current round. The final loss is the average of each round’s loss. Next, by adding conditional random fields for sequence labeling, we ensure that the model pays more attention to the overall labeling results. In addition, we use the focal loss to solve the problem of category imbalance caused by the fact that most words in text error correction do not need error correction. Furthermore, the experiments on NLPCC 2018 Task 2 show that our method outperforms prior work by up to 2% on the F0.5 score, which verifies the efficiency of iterative training on the Chinese GEC model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Text Readability and Simplification · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China
References (30)
Show source-provided references
1. Yuan, Z., and Briscoe, T. (2016, January 12–17). Grammatical error correction using neural machine translation. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1042 ↗
2. Awasthi, A., Sarawagi, S., Goyal, R., Ghosh, S., and Piratla, V. (2019, January 3–7). Parallel Iterative Edit Models for Local Sequence Transduction. Proceedings of the 2019 Conference on Empirical Methods in Natural Language, Hong Kong, China.10.18653/v1/D19-1435 ↗
3. Omelianchuk, K., Atrasevych, V., Chernodub, A., and Skurzhanskyi, O. (2020, January 10). GECToR–Grammatical Error Correction: Tag, Not Rewrite. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Online.10.18653/v1/2020.bea-1.16 ↗
4. Goodman, S., Ding, N., and Soricut, R. (2020, January 16–20). TeaForN: Teacher-Forcing with N-grams. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.702 ↗
5. Lafferty, J.D., McCallum, A., and Pereira, F.C. (July, January 28). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williamstown, MA, USA.
6. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
7. Zhao, Y., Jiang, N., Sun, W., and Wan, X. (2018, January 13–17). Overview of the nlpcc 2018 shared task: Grammatical error correction. Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing, Qingdao, China.10.1007/978-3-319-99501-4_41 ↗
8. Sentence correction incorporating relative position and parse template language models
9. Yu, C.H., and Chen, H.H. (2012, January 8–15). Detecting word ordering errors in Chinese sentences for learning Chinese as a foreign language. Proceedings of the COLING 2012, Mumbai, India.
10. Yu, L.C., Lee, L.H., and Chang, L.P. (2014, January 30). Overview of grammatical error diagnosis for learning Chinese as a foreign language. Proceedings of the 1st Workshop on Natural Language Processing Techniques for Educational Applications (NLPTEA-14), Nara, Japan.
11. Hu, Q., Zhang, Y., Liu, F., and Gu, Y. (2018, January 19). Ling@ cass solution to the nlp-tea cged shared task 2018. Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, Melbourne, Australia.10.18653/v1/W18-3709 ↗
12. Lichtarge, J., Alberti, C., Kumar, S., Shazeer, N., Parmar, N., and Tong, S. (2019). Corpora Generation for Grammatical Error Correction. arXiv.10.18653/v1/N19-1333 ↗
13. Ren, H., Yang, L., and Xun, E. (2018, January 13–17). A sequence to sequence learning for Chinese grammatical error correction. Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing, Qingdao, China.10.1007/978-3-319-99501-4_36 ↗
14. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y.N. (2017, January 6–11). Convolutional sequence to sequence learning. Proceedings of the International Conference on Machine Learning (PMLR 2017), Sydney, Australia.
15. Chen, M., Ge, T., Zhang, X., Wei, F., and Zhou, M. (2020, January 16–20). Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.581 ↗
16. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
17. Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., and Liu, T. (2020, January 26–30). Incorporating BERT into Neural Machine Translation. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
18. Wang, H., Kurosawa, M., Katsumata, S., and Komachi, M. (2020, January 5–6). Chinese Grammatical Correction Using BERT-based Pre-trained Model. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Suzhou, China.
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. arXiv.
20. Kaneko, M., Mita, M., Kiyono, S., Suzuki, J., and Inui, K. (2020, January 5–10). Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.391 ↗
21. Li, P., and Shi, S. (2021). Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction. arXiv.10.18653/v1/2021.acl-long.385 ↗
22. Malmi, E., Krause, S., Rothe, S., Mirylenka, D., and Severyn, A. (2019, January 3–7). Encode, Tag, Realize: High-Precision Text Editing. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1510 ↗
23. Rao, G., Yang, E., and Zhang, B. (2020, January 4). Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis. Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, Suzhou, China.
24. Liang, D., Zheng, C., Guo, L., Cui, X., Xiong, X., Rong, H., and Dong, J. (2020, January 4). BERT enhanced neural machine translation and sequence tagging model for Chinese grammatical error diagnosis. Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, Suzhou, China.
25. Zhao, Z., and Wang, H. (2020, January 7–12). MaskGEC: Improving neural grammatical error correction via dynamic masking. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i01.5476 ↗
26. Tang, Z., Ji, Y., Zhao, Y., and Li, J. (2021, January 13–15). Chinese Grammatical Error Correction enhanced by Data Augmentation from Word and Character Levels. Proceedings of the 20th Chinese National Conference on Computational Linguistics, Hohhot, China.
27. Sun, Z., Li, Z., Wang, H., He, D., Lin, Z., and Deng, Z. (2019). Fast structured decoding for sequence models. arXiv.
28. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020). Revisiting Pre-Trained Models for Chinese Natural Language Processing. arXiv.10.18653/v1/2020.findings-emnlp.58 ↗
29. Fu, K., Huang, J., and Duan, Y. (2018, January 26–30). Youdao’s Winning Solution to the NLPCC-2018 Task 2 Challenge: A Neural Machine Translation Approach to Chinese Grammatical Error Correction. Proceedings of the 7th CCF International Conference (NLPCC 2018), Hohhot, China.10.1007/978-3-319-99495-6_29 ↗
30. Zhou, J., Li, C., Liu, H., Bao, Z., Xu, G., and Li, L. (2018, January 13–17). Chinese grammatical error correction using statistical and neural models. Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing, Qingdao, China.10.1007/978-3-319-99501-4_10 ↗