What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams
Source abstract
Abstract
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Sentiment Analysis and Opinion Mining
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Artificial Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430074, China
References (42)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, MN, USA.
2. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
3. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv.
4. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016, January 1–5). SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, TX, USA.10.18653/v1/D16-1264 ↗
5. Rajpurkar, P., Jia, R., and Liang, P. (2018, January 15–20). Know What You Don’t Know: Unanswerable Questions for SQuAD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-2124 ↗
6. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017, January 7–11). RACE: Large-scale ReAding Comprehension Dataset From Examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark.10.18653/v1/D17-1082 ↗
7. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C.D. (November, January 31). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium.10.18653/v1/D18-1259 ↗
8. Gao, S., Agarwal, S., Jin, D., Chung, T., and Hakkani-Tur, D. (2020, January 9). From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, Association for Computational Linguistics, Online.10.18653/v1/2020.nlp4convai-1.10 ↗
9. Dunn, M., Sagun, L., Higgins, M., Güney, V.U., Cirik, V., and Cho, K. (2017). SearchQA: A New QA Dataset Augmented with Context from a Search Engine. arXiv.
10. Chen, D., Fisch, A., Weston, J., and Bordes, A. Reading Wikipedia to Answer Open-Domain Questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, BC, Canada.
11. Clark, C., and Gardner, M. (2018, January 15–20). Simple and Effective Multi-Paragraph Reading Comprehension. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1078 ↗
12. Wang, Z., Ng, P., Ma, X., Nallapati, R., and Xiang, B. (2019). Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. arXiv.10.18653/v1/D19-1599 ↗
13. Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., and Xiong, C. (2020). Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering. arXiv.
14. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (August, January 30). TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, BC, Canada.
15. Dhingra, B., Mazaitis, K., and Cohen, W.W. (2017). Quasar: Datasets for Question Answering by Search and Reading. arXiv.
16. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv.
17. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (November, January 31). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium.10.18653/v1/D18-1260 ↗
18. Coqa: A conversational question answering challenge10.1162/tacl_a_00266 ↗
19. Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., and Jiang, J. (2018, January 2–7). R3: Reinforced Ranker-Reader for Open-Domain Question Answering. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12053 ↗
20. Das, R., Dhuliawala, S., Zaheer, M., and McCallum, A. (2019). Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. arXiv.
21. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension10.1162/tacl_a_00305 ↗
22. Zhang, X., Wu, J., He, Z., Liu, X., and Su, Y. (2018, January 2–7). Medical exam question answering with large-scale reading comprehension. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11970 ↗
23. Yu, W., Jiang, Z.H., Dong, Y., and Feng, J. (2020). ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. arXiv.
24. Abacha, A.B., Agichtein, E., Pinter, Y., and Demner-Fushman, D. (2017). Overview of the Medical Question Answering Task at TREC 2017 LiveQA, TREC.
25. Abacha, A.B., Mrabet, Y., Sharp, M., Goodwin, T.R., Shooshan, S.E., and Demner-Fushman, D. (2019). Bridging the Gap between Consumers’ Medication Questions and Trusted Answers, MedInfo.
26. Ben Abacha, A., Shivade, C., and Demner-Fushman, D. (2019, January 1). Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy.10.18653/v1/W19-5039 ↗
27. Abacha, A.B., and Demner-Fushman, D. (2019). A question-entailment approach to question answering. BMC Bioinf., 20.10.1186/s12859-019-3119-4 ↗
28. Consumer health information and question answering: Helping consumers find answers to their health-related information needs10.1093/jamia/ocz152 ↗
29. Nentidis, A., Krithara, A., Bougiatiotis, K., Krallinger, M., Rodriguez-Penagos, C., Villegas, M., and Paliouras, G. (2020, January 22–25). Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. Proceedings of the International Conference of the Cross-Language Evaluation Forum for European Languages, Thessaloniki, Greece.10.1007/978-3-030-58219-7_16 ↗
30. Pampari, A., Raghavan, P., Liang, J., and Peng, J. (November, January 31). emrQA: A Large Corpus for Question Answering on Electronic Medical Records. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium.10.18653/v1/D18-1258 ↗
31. Ha, L.A., and Yaneva, V. (2019, January 2–4). Automatic Question Answering for Medical MCQs: Can It go Further than Information Retrieval?. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), Varna, Bulgaria.10.26615/978-954-452-056-4_049 ↗
32. oLMpics—On what Language Model Pre-training Captures10.1162/tacl_a_00342 ↗
33. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.H., and Riedel, S. (2019). Language Models as Knowledge Bases?. arXiv.10.18653/v1/D19-1250 ↗
34. Clark, P., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P., and Khashabi, D. (2016, January 12–17). Combining retrieval, statistics, and inference to answer elementary science questions. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10325 ↗
35. Jin, D., Gao, S., Kao, J.Y., Chung, T., and Hakkani-Tür, D.Z. (2019). MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension. arXiv.10.1609/aaai.v34i05.6310 ↗
36. Yan, M., Zhang, H., Jin, D., and Zhou, J.T. (2020, January 5–10). Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.654 ↗
37. Moen, S., and Ananiadou, T.S.S. (2013). Distributional semantics resources for biomedical text processing. Proc. LBM, 39–44.
38. Enriching Word Vectors with Subword Information10.1162/tacl_a_00051 ↗
39. Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z., Wang, S., and Hu, G. (2019). Pre-Training with Whole Word Masking for Chinese BERT. arXiv.
40. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., and Kang, J. (2019). BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.10.1093/bioinformatics/btz682 ↗
41. Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jin, D., Naumann, T., and McDermott, M. (2019, January 7). Publicly Available Clinical BERT Embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop, Association for Computational Linguistics, Minneapolis, MN, USA.10.18653/v1/W19-1909 ↗
42. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N.A. (2020). Do not Stop Pretraining: Adapt Language Models to Domains and Tasks. arXiv.10.18653/v1/2020.acl-main.740 ↗