Automated Assessment of Inferences Using Pre-Trained Language Models
Source abstract
Abstract
Inference plays a key role in reading comprehension. However, assessing inference in reading is a complex process that relies on the judgment of trained experts. In this study, we explore objective and automated methods for assessing inference in readers’ responses using natural language processing. Specifically, classifiers were trained to detect inference from a pair of input texts and reader responses by fine-tuning three widely used pre-trained language models. The effects of the model size and pre-training strategy on the accuracy of inference classification were investigated. The highest F1 score of 0.92 was achieved via fine-tuning the robustly optimized 12-layer BERT model (RoBERTa-base). Fine-tuning the larger 24-layer model (RoBERTa-large) did not improve the classification accuracy. Error analysis provides insight into the relative difficulty of classifying inference subtypes. The proposed method demonstrates the feasibility of the automated quantification of inference during reading, and offers potential to facilitate individualized reading instructions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Text Readability and Simplification · Topic Modeling · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Soongsil University, Seoul 06978, Republic of Korea
References (35)
Show source-provided references
1. Strzalkowski, T. (1999). Natural Language Information Retrieval, Springer.10.1007/978-94-017-2388-6 ↗
2. Yunshi, L., He, G., Jiang, J., Jiang, J., Zhao, W.X., and Wen, J.-R. (2021, January 19–27). A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence Survey Track, Montreal, QC, Canada.
3. More than a feeling: Accuracy and application of sentiment analysis10.1016/j.ijresmar.2022.05.005 ↗
4. A survey of sentiment analysis from social media data10.1109/TCSS.2019.2956957 ↗
5. Ilias, C., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D., and Aletras, N. (2022, January 22–27). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.
6. Zhuang, L., Huang, D., Huang, K., Li, Z., and Zhao, J. (2021, January 7–15). Finbert: A pre-trained financial language representation model for financial text mining. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, Yokohama, Japan.
7. Automated essay scoring systems: A systematic literature review10.1007/s10462-021-10068-2 ↗
8. Snowling, M.J., and Hulme, C. (2005). The Science of Reading: A Handbook, Blackwell Publishing.10.1111/b.9781405114882.2005.x ↗
9. The science of reading progresses: Communicating advances beyond the simple view of reading10.1002/rrq.411 ↗
10. Riazi, A.M. (2016). The Routledge Encyclopedia of Research Methods in Applied Linguistics, Routledge.10.4324/9781315656762 ↗
11. Gallagher, K. (2009). Readicide: How Schools Are Killing Reading and What You Can Do about It, Routledge.
12. A machine learning approach to reading level assessment10.1016/j.csl.2008.04.003 ↗
13. Kincaid, J.P., Fishburne, R.P., Rodgers, R.L., and Chisson, B.S. (1975). Derivation of New Readability Formulas for Navy Enlisted Personnel, US Naval Air Station. Research Branch Report 8-75.
14. Boonthum-Denecke, C., McCarthy, P., Lamkin, T., Jackson, G.T., Magliano, J.P., and McNamara, D.S. (2011, January 18–20). Automatic natural language processing and the detection of reading skills and reading comprehension. Proceedings of the Twenty-Fourth International FLAIRS Conference, Palm Beach, FL, USA.
15. Landauer, T., McNamara, D., Dennis, S., and Kintsch, W. (2013). Handbook of Latent Semantic Analysis, Psychology Press.
16. Allen, L.K., Snow, E.L., and McNamara, D.S. (2015, January 16–20). Are you reading my mind? Modeling students’ reading comprehension skills with Natural Language Processing techniques. Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, Poughkeepsie, NY, USA.10.1145/2723576.2723617 ↗
17. McNamara, D.S., Graesser, A.C., McCarthy, P., and Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix, Cambridge University Press.10.1017/CBO9780511894664 ↗
18. MacGinitie, W.H. (1989). Gates-MacGinitie Reading Tests, Houghton Mifflin.
19. Yuill, N., and Oakhill, J. (2010). Children’s Problems in Text Comprehension: An Experimental Investigation, Cambridge University Press.
20. Inference making and its relation to comprehension failure in young children10.1023/A:1008084120205 ↗
21. Assessing children’s inference generation: What do tests of reading comprehension measure?10.1348/000709904X22674 ↗
22. Ericsson, K.A., and Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data, MIT Press.10.7551/mitpress/5657.001.0001 ↗
23. Making the right connections: Differential effects of reading intervention for subgroups of comprehenders10.1016/j.lindif.2011.11.017 ↗
24. Development of a new reading comprehension assessment: Identifying comprehension differences among readers10.1016/j.lindif.2014.03.003 ↗
25. Bowles, M.A. (2010). The Think-Aloud Controversy in Second Language Research, Routledge.10.4324/9780203856338 ↗
26. Recent advances in natural language processing via large pre-trained language models: A survey
27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
28. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv.
29. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv.
30. Park, S., Moon, J., Kim, S., Cho, W.I., Han, J., Park, J., Song, C., Kim, J., Song, Y., and Oh, T. (2021). KLUE: Korean language understanding evaluation. arXiv.
31. Neural machine translation of rare words with subword units10.18653/v1/P16-1162 ↗
32. (2024, March 01). BERT-Base Model Pretrained on KLUE. Available online: https://huggingface.co/klue/bert-base.
33. (2024, March 01). RoBERTa-Base Model Pretrained on KLUE. Available online: https://huggingface.co/klue/roberta-base.
34. (2024, March 01). RoBERTa-Large Model Pretrained on KLUE. Available online: https://huggingface.co/klue/roberta-large.
35. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.