A Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Source abstract
Abstract
Machine Reading Comprehension (MRC) is a challenging Natural Language Processing (NLP) research field with wide real-world applications. The great progress of this field in recent years is mainly due to the emergence of large-scale datasets and deep learning. At present, a lot of MRC models have already surpassed human performance on various benchmark datasets despite the obvious giant gap between existing MRC models and genuine human-level reading comprehension. This shows the need for improving existing datasets, evaluation metrics, and models to move current MRC models toward “real” understanding. To address the current lack of comprehensive survey of existing MRC tasks, evaluation metrics, and datasets, herein, (1) we analyze 57 MRC tasks and datasets and propose a more precise classification method of MRC tasks with 4 different attributes; (2) we summarized 9 evaluation metrics of MRC tasks, 7 attributes and 10 characteristics of MRC datasets; (3) We also discuss key open issues in MRC research and highlighted future research directions. In addition, we have collected, organized, and published our data on the companion website where MRC researchers could directly access each MRC dataset, papers, baseline projects, and the leaderboard.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Chengdu Institute of Computer Application, Chinese Academy of Sciences, Chengdu 610041, China
University of Chinese Academy of Sciences, Beijing 100049, China
Department of Computer Science and Engineering, Chengdu Neusoft University, Chengdu 611844, China
School of Mechanical Engineering, Guizhou University, Guiyang 550025, China
College of Big Data Statistics, GuiZhou University of Finance and Economics, Guiyang 550025, China
Department of Computer Science and Engineering, University of South Carolina, Columbia, SC 29208, USA
Funding
National Important Project of China · Guizhou Province Science and Technology Project · National Science Foundation
References (100+)
Show source-provided references
1. Recent trends in deep learning based natural language processing10.1109/MCI.2018.2840738 ↗
2. DeepPatent: Patent classification with convolutional neural networks and word embedding10.1007/s11192-018-2905-5 ↗
3. Li, Q., Li, S., Hu, J., Zhang, S., and Hu, J. (2018). Tourism Review Sentiment Classification Using a Bidirectional Recurrent Neural Network with an Attention Mechanism and Topic-Enriched Word Vectors. Sustainability, 10.10.3390/su10093313 ↗
4. Reading pictures for story comprehension requires mental imagery skills10.3389/fpsyg.2016.01630 ↗
5. Sugawara, S., Stenetorp, P., Inui, K., and Aizawa, A. (2019). Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets. arXiv.10.1609/aaai.v34i05.6422 ↗
6. Zhang, X., Yang, A., Li, S., and Wang, Y. (2019). Machine reading comprehension: A literature review. arXiv.
7. Baradaran, R., Ghiasi, R., and Amirkhani, H. (2020). A Survey on Machine Reading Comprehension Systems. arXiv.
8. Gupta, S., and Rawat, B.P.S. (2020). Conversational Machine Comprehension: A Literature Review. arXiv.10.18653/v1/2020.coling-main.247 ↗
9. Gao, J., Galley, M., and Li, L. (2018, January 15–20). Neural Approaches to Conversational AI. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Melbourne, Australia.10.18653/v1/P18-5002 ↗
10. Lehnert, W.G. (1977). The Process of Question Answering. [Ph.D. Thesis, Yale Univ New Haven Conn Dept Of Computer Science].
11. Hirschman, L., Light, M., Breck, E., and Burger, J.D. (1999, January 20–26). Deep Read: A Reading Comprehension System. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, College Park, MD, USA.10.3115/1034678.1034731 ↗
12. Riloff, E., and Thelen, M. (2000, January 2–7). A Rule-Based Question Answering System for Reading Comprehension Tests. Proceedings of the ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems, New Orleans, LA, USA.10.3115/1117595.1117598 ↗
13. Charniak, E., Altun, Y., de Salvo Braz, R., Garrett, B., Kosmala, M., Moscovich, T., Pang, L., Pyo, C., Sun, Y., and Wy, W. (May, January 29). Reading Comprehension Programs in a Statistical-Language-Processing Class. Proceedings of the ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems, Seattle, WA, USA.10.3115/1117595.1117596 ↗
14. Chen, D. (2018). Neural Reading Comprehension and Beyond. [Ph.D. Thesis, Stanford University].
15. Richardson, M., Burges, C.J., and Renshaw, E. (2013, January 18–21). MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.
16. Wang, H., Bansal, M., Gimpel, K., and McAllester, D. (2015, January 26–31). Machine comprehension with syntax, frames, and semantics. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Beijing, China.10.3115/v1/P15-2115 ↗
17. Sachan, M., Dubey, K., Xing, E., and Richardson, M. (2015, January 26–31). Learning answer-entailing structures for machine comprehension. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1024 ↗
18. Narasimhan, K., and Barzilay, R. (2015, January 26–31). Machine comprehension with discourse relations. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1121 ↗
19. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
20. Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2016). Bidirectional attention flow for machine comprehension. arXiv.
21. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
22. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
23. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
24. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., and Le, Q.V. (2019, January 8–14). Xlnet: Generalized autoregressive pretraining for language understanding. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
25. Grail, Q., and Perez, J. (2018). ReviewQA: A relational aspect-based opinion reading dataset. arXiv.
26. Qiu, B., Chen, X., Xu, J., and Sun, Y. (2019). A Survey on Neural Machine Reading Comprehension. arXiv.
27. Neural machine reading comprehension: Methods and trends10.3390/app9183698 ↗
28. Hill, F., Bordes, A., Chopra, S., and Weston, J. (2016, January 2–4). The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.
29. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. (2016, January 2–4). Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.
30. Jia, R., and Liang, P. (2017, January 9–11). Adversarial Examples for Evaluating Reading Comprehension Systems. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1215 ↗
31. Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. (2017, January 21–26). Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.571 ↗
32. Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., and Fidler, S. (2016, January 27–30). Movieqa: Understanding stories in movies through question-answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.501 ↗
33. Iyyer, M., Manjunatha, V., Guha, A., Vyas, Y., Boyd-Graber, J., Daume, H., and Davis, L.S. (2017, January 21–26). The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.686 ↗
34. Yagcioglu, S., Erdem, A., Erdem, E., and Ikizler-Cinbis, N. (November, January 31). RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1166 ↗
35. Chen, D., Fisch, A., Weston, J., and Bordes, A. (August, January 30). Reading Wikipedia to Answer Open-Domain Questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
36. Hu, M. (2019). Research on Machine Reading Comprehension and Textual Question Answering. [Ph.D. Thesis, National University of Defense Technology of China].
37. Vanderwende, L. (2007, January 26–28). Answering and Questioning for Machine Reading. Proceedings of the AAAI 2007 Spring Symposium, Palo Alto, CA, USA.
38. Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., and Riedel, S. (November, January 31). Interpretation of Natural Language Rules in Conversational Machine Reading. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1233 ↗
39. Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv.
40. Liu, X., Shen, Y., Duh, K., and Gao, J. (2018). Stochastic Answer Networks for Machine Reading Comprehension. arXiv.10.18653/v1/P18-1157 ↗
41. Liu, X., Li, W., Fang, Y., Kim, A., Duh, K., and Gao, J. (2018). Stochastic Answer Networks for SQuAD 2.0. arXiv.
42. Liu, X., Duh, K., and Gao, J. (2018). Stochastic Answer Networks for Natural Language Inference. arXiv.
43. Yin, Y., Song, Y., and Zhang, M. (2017, January 9–11). Document-Level Multi-Aspect Sentiment Classification as Machine Comprehension. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1217 ↗
44. Asai, A., Eriguchi, A., Hashimoto, K., and Tsuruoka, Y. (2018). Multilingual Extractive Reading Comprehension by Runtime Machine Translation. arXiv.
45. Zhang, Z., Wu, Y., Zhou, J., Duan, S., and Wang, R. (2020, January 7–12). SG-Net: Syntax-Guided Machine Reading Comprehension. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, NY, USA.
46. Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., and Gao, J. (2020). Deep Learning Based Text Classification: A Comprehensive Review. arXiv.10.1145/3439726 ↗
47. Coqa: A conversational question answering challenge10.1162/tacl_a_00266 ↗
48. Onishi, T., Wang, H., Bansal, M., Gimpel, K., and McAllester, D. (2016, January 1–5). Who did What: A Large-Scale Person-Centered Cloze Dataset. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1241 ↗
49. Constructing datasets for multi-hop reading comprehension across documents10.1162/tacl_a_00021 ↗
50. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. (2019). DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. arXiv.
51. Baudiš, P., and Šedivỳ, J. (2015). Modeling of the question answering task in the yodaqa system. International Conference of the Cross-Language Evaluation Forum for European Languages, Springer.10.1007/978-3-319-24027-5_20 ↗
52. Yang, Y., Yih, W.T., and Meek, C. (2015, January 17–21). Wikiqa: A challenge dataset for open-domain question answering. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1237 ↗
53. Miller, A.H., Fisch, A., Dodge, J., Karimi, A.H., Bordes, A., and Weston, J. (2016). Key-Value Memory Networks for Directly Reading Documents (EMNLP16). arXiv.10.18653/v1/D16-1147 ↗
54. Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, K. (August, January 30). NewsQA: A Machine Comprehension Dataset. Proceedings of the 2nd Workshop on Representation Learning for NLP, Vancouver, BC, Canada.
55. Boleda, G., Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q.N., Bernardi, R., Pezzelle, S., Baroni, M., and Fernandez, R. (2016, January 7–12). The LAMBADA dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1144 ↗
56. Hewlett, D., Lacoste, A., Jones, L., Polosukhin, I., Fandrianto, A., Han, J., Kelcey, M., and Berthelot, D. (2016, January 7–12). WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1145 ↗
57. Bajgar, O., Kadlec, R., and Kleindienst, J. (2016). Embracing data abundance: Booktest dataset for reading comprehension. arXiv.
58. Soricut, R., and Ding, N. (2016). Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors. arXiv.
59. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (August, January 30). TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
60. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017, January 9–11). RACE: Large-scale ReAding Comprehension Dataset From Examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1082 ↗
61. Dhingra, B., Mazaitis, K., and Cohen, W.W. (2017). Quasar: Datasets for Question Answering by Search and Reading. arXiv.
62. Dunn, M., Sagun, L., Higgins, M., Guney, V.U., Cirik, V., and Cho, K. (2017). Searchqa: A new q&a dataset augmented with context from a search engine. arXiv.
63. The narrativeqa reading comprehension challenge10.1162/tacl_a_00023 ↗
64. Welbl, J., Liu, N.F., and Gardner, M. (2017, January 7). Crowdsourcing Multiple Choice Science Questions. Proceedings of the 3rd Workshop on Noisy User-generated Text, Copenhagen, Denmark.10.18653/v1/W17-4413 ↗
65. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.T., Choi, Y., Liang, P., and Zettlemoyer, L. (November, January 31). QuAC: Question Answering in Context. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1241 ↗
66. Rajpurkar, P., Jia, R., and Liang, P. (2018, January 15–20). Know What You Don’t Know: Unanswerable Questions for SQuAD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia.10.18653/v1/P18-2124 ↗
67. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C.D. (November, January 31). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1259 ↗
68. Saha, A., Aralikatte, R., Khapra, M.M., and Sankaranarayanan, K. (2018, January 15–20). DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1156 ↗
69. Xie, Q., Lai, G., Dai, Z., and Hovy, E. (November, January 31). Large-scale Cloze Test Dataset Created by Teachers. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1257 ↗
70. Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., and Van Durme, B. (2018). Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv.
71. Šuster, S., and Daelemans, W. (2018, January 1–6). CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1140 ↗
72. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv.
73. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. arXiv.10.18653/v1/D18-1260 ↗
74. Khot, T., Sabharwal, A., and Clark, P. (2018, January 2–7). Scitail: A textual entailment dataset from science question answering. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12022 ↗
75. Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. (2018, January 1–6). Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1023 ↗
76. Can Machines Learn to Comprehend Scientific Literature?10.1109/ACCESS.2019.2891666 ↗
77. Hong, Y., Wang, J., Zhang, X., and Wu, Z. (2018, January 2–7). Learning to Read Academic Papers. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, LA, USA.
78. Ostermann, S., Modi, A., Roth, M., Thater, S., and Pinkal, M. (2018, January 7–12). MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
79. Dalvi, B., Huang, L., Tandon, N., Yih, W.T., and Clark, P. (2018, January 1–6). Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1144 ↗
80. Natural questions: A benchmark for question answering research10.1162/tacl_a_00276 ↗
81. Dream: A challenge data set and models for dialogue-based reading comprehension10.1162/tacl_a_00264 ↗
82. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2018, January 1–6). CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), New Orleans, LA, USA.
83. Lin, C.Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries, Association for Computational Linguistics. Text Summarization Branches Out.
84. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
85. Banerjee, S., and Lavie, A. (2005, January 25–30). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
86. Denkowski, M., and Lavie, A. (2010, January 15–16). METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support for Five Target Languages. Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, Uppsala, Sweden.
87. Denkowski, M., and Lavie, A. (2011, January 30–31). Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, UK.
88. Van Rijsbergen, C.J. (1979). Information Retrieval, Butterworth.
89. Yuen, M.C., King, I., and Leung, K.S. (2011, January 9–11). A survey of crowdsourcing systems. Proceedings of the 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, Boston, MA, USA.10.1109/PASSAT/SocialCom.2011.203 ↗
90. Sugawara, S., and Aizawa, A. (2016, January 5). An analysis of prerequisite skills for reading comprehension. Proceedings of the Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods, Austin, TX, USA.10.18653/v1/W16-6001 ↗
91. Sugawara, S., Kido, Y., Yokono, H., and Aizawa, A. (August, January 30). Evaluation metrics for machine reading comprehension: Prerequisite skills and readability. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
92. Yan, M., Xia, J., Wu, C., Bi, B., Zhao, Z., Zhang, J., Si, L., Wang, R., Wang, W., and Chen, H. (February, January 27). A deep cascade model for multi-document reading comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
93. Wang, Y., Liu, K., Liu, J., He, W., Lyu, Y., Wu, H., Li, S., and Wang, H. (2018, January 15–20). Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1178 ↗
94. Bowman, S.R., Angeli, G., Potts, C., and Manning, C.D. (2015, January 17–21). A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal.10.18653/v1/D15-1075 ↗
95. Speer, R., Chin, J., and Havasi, C. (2017, January 4–9). Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11164 ↗
96. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–13). Vqa: Visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.279 ↗
97. Hart, M. (2020, October 24). Project Gutenberg. Available online: https://www.gutenberg.org.
98. NIST (2020, October 24). The Text REtrieval Conference (TREC), Available online: https://trec.nist.gov/.
99. Raison, M., Mazaré, P.E., Das, R., and Bordes, A. (2018). Weaver: Deep co-encoding of questions and documents for machine reading. arXiv.
100. Sampath, R., and Ma, P. (2020, October 24). QA with Wiki: Improving Information Retrieval and Machine Comprehension. Available online: https://web.stanford.edu/class/cs224n/reports/custom/15737727.pdf.