AT-CRF: A Chinese Reading Comprehension Algorithm Based on Attention Mechanism and Conditional Random Fields
Source abstract
Abstract
Machine reading comprehension (MRC) is an important research topic in the field of Natural Language Processing (NLP). However, traditional MRC models often face challenges of information loss, lack of capability to retain long-distance dependence, and inability to deal with unanswerable questions where answers are not available in given texts. In this paper, a Chinese reading comprehension algorithm, called the Attention and Conditional Random Filed (AT-CRF) Reader, is proposed to address the above challenges. Firstly, RoBERTa, a pre-trained language model, is introduced to obtain the embedding representations of input. Then, a depthwise separable convolution neural network and attention mechanisms are used to replace the recurrent neural network for encoding. Next, the attention flow and self-attention mechanisms are used to obtain the context–query internal relation. Finally, the conditional random field is used to handle unanswerable questions and predict the correct answer. Experiments were conducted on the two Chinese machine reading comprehension datasets, CMRC2018 and DuReader-checklist, and the results showed that, compared with the baseline model, the F1 scores achieved by our AT-CRF Reader model has improved by 2.65% and 2.68%, and the EM values increased by 4.45% and 3.88%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical Engineering, Southwest Minzu University, Chengdu 610041, China
Computer Science, University of Hull, Hull HU6 7RX, UK
Funding
Southwest Minzu University Graduate Innovative Research Project · Sichuan Science and Technology Applied Foundation Project · National Natural Science Foundation of China · Fundamental Research Funds for the Central Universities, Southwest Minzu University
References (30)
Show source-provided references
1. Liu, S., Zhang, X., Zhang, S., Wang, H., and Zhang, W. (2019). Neural machine reading comprehension: Methods and trends. Appl. Sci., 9.10.3390/app9183698 ↗
2. Review of Conversational Machine Reading Comprehension
3. Kenton, J.D.M.W.C., and Toutanova, L.K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
4. Chi, P., Chung, P., Wu, T., Hsieh, C., Chen, Y., Li, S., and Lee, H. (2021, January 19–22). Audio albert: A lite Bert for self-supervised learning of audio representation. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Virtual.10.1109/SLT48900.2021.9383575 ↗
5. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
6. Review of Pre-training Techniques for Natural Language Processing
7. Modeling multi-prototype Chinese word representation learning for word similarity10.1007/s40747-021-00482-y ↗
8. Extracting Knowledge Entities from Sci-Tech Intelligence Resources Based on BiLSTM and Conditional Random Field10.1587/transinf.2020BDP0007 ↗
9. Cui, Y., Liu, T., Che, W., Xiao, L., Chen, Z., Ma, W., Wang, S., and Hu, G. (2019, January 3–7). A Span-Extraction Dataset for Chinese Machine Reading Comprehension. Proceedings of the EMNLP/IJCNLP, Hong Kong, China.10.18653/v1/D19-1600 ↗
10. He, W., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H., and She, Q. (2018, January 19). DuReader: A Chinese Machine Reading Comprehension Dataset from Real-World Applications. Proceedings of the QA@ ACL, Melbourne, Australia.10.18653/v1/W18-2605 ↗
11. Riloff, E., and Thelen, M. (2000, January 4). A rule-based question answering system for reading comprehension tests. Proceedings of the ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems, Seattle, WA, USA.10.3115/1117595.1117598 ↗
12. Hirschman, L., Light, M., Breck, E., and Burger, J.D. (1999, January 20–26). Deep read: A reading comprehension system. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, College Park, MD, USA.10.3115/1034678.1034731 ↗
13. Chen, D., Bolton, J., and Manning, C.D. (2016). A thorough examination of the cnn/daily mail reading comprehension task. arXiv.10.18653/v1/P16-1223 ↗
14. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
15. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for SQuAD. arXiv.10.18653/v1/P18-2124 ↗
16. Cui, Y., Chen, Z., Wei, S., Wang, S., Liu, T., and Hu, G. (2016). Attention-over-attention neural networks for reading comprehension. arXiv.10.18653/v1/P17-1055 ↗
17. Wang, S., and Jiang, J. (2016). Machine comprehension using match-lstm and answer pointer. arXiv.
18. Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2016). Bidirectional attention flow for machine comprehension. arXiv.
19. Wang, W., Yang, N., Wei, F., Chang, B., and Zhou, M. (August, January 30). Gated self-matching networks for reading comprehension and question answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
20. Attention is all you need
21. Cui, Y., Liu, T., Chen, Z., Wang, S., and Hu, G. (2016). Consensus attention-based neural networks for Chinese reading comprehension. arXiv.10.18653/v1/P17-1055 ↗
22. D-Reader: A Reading Comprehension Model by Full-text Prediction
23. Pre-training with whole word masking for chinese bert10.1109/TASLP.2021.3124365 ↗
24. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019). ERNIE: Enhanced Language Representation with Informative Entities. arXiv.10.18653/v1/P19-1139 ↗
25. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv.
26. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020). Revisiting pre-trained models for Chinese natural language processing. arXiv.10.18653/v1/2020.findings-emnlp.58 ↗
27. Frame-based neural network for machine reading comprehension10.1016/j.knosys.2021.106889 ↗
28. Sun, Z., Li, X., Sun, X., Meng, Y., Ao, X., He, Q., Wu, F., and Li, J. (2021). Chinesebert: Chinese pretraining enhanced by glyph and pinyin information. arXiv.10.18653/v1/2021.acl-long.161 ↗
29. Hidden markov model based named entity recognition tool
30. Xie, C., Chen, D., Shi, H., and Fan, M. (2021, January 6–7). Attention-Based Bidirectional Long Short Term Memory Networks Combine with Phrase Convolution Layer for Relation Extraction. Proceedings of the 2021 5th SLAAI International Conference on Artificial Intelligence (SLAAI-ICAI), Kelaniya, Sri Lanka.10.1109/SLAAI-ICAI54477.2021.9664707 ↗