DATLMedQA: A Data Augmentation and Transfer Learning Based Solution for Medical Question Answering
Source abstract
Abstract
With the outbreak of COVID-19 that has prompted an increased focus on self-care, more and more people hope to obtain disease knowledge from the Internet. In response to this demand, medical question answering and question generation tasks have become an important part of natural language processing (NLP). However, there are limited samples of medical questions and answers, and the question generation systems cannot fully meet the needs of non-professionals for medical questions. In this research, we propose a BERT medical pretraining model, using GPT-2 for question augmentation and T5-Small for topic extraction, calculating the cosine similarity of the extracted topic and using XGBoost for prediction. With augmentation using GPT-2, the prediction accuracy of our model outperforms the state-of-the-art (SOTA) model performance. Our experiment results demonstrate the outstanding performance of our model in medical question answering and question generation tasks, and its great potential to solve other biomedical question answering challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Biomedical Text Mining and Ontologies · Artificial Intelligence in Healthcare and Education
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Informatics, King’s College London, Strand, London WC2R 2LS, UK
Department of Computer Science, School of Engineering and Applied Science, Gonzaga University, Spokane, WA 99258, USA
References (46)
Show source-provided references
1. Chinese Public’s Attention to the COVID-19 Epidemic on Social Media: Observational Descriptive Study10.2196/18825 ↗
2. COVID-19 randomized controlled trials in medRxiv and PubMed10.1016/j.ejim.2020.09.019 ↗
3. Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., and Lu, X. (2019). PubMedQA: A Dataset for Biomedical Research Question Answering. arXiv.10.18653/v1/D19-1259 ↗
4. Ong, E., Wong, M.U., Huffman, A., and He, Y. (2020). COVID-19 coronavirus vaccine design using reverse vaccinology and machine learning. bioRxiv.10.1101/2020.03.20.000141 ↗
5. Mahase, E. (2020). COVID-19: WHO declares pandemic because of “alarming levels” of spread, severity, and inaction. BMJ, 368.10.1136/bmj.m1036 ↗
6. Surita, G., Nogueira, R., and Lotufo, R. (2020). Can questions summarize a corpus? Using question generation for characterizing COVID-19 research. arXiv.
7. Yadav, S., Gupta, D., Abacha, A., and Demner-Fushman, D. (2021). Question-aware Transformer Models for Consumer Health Question Summarization. arXiv.10.1016/j.jbi.2022.104040 ↗
8. CIDO, a community-based ontology for coronavirus disease knowledge and data integration, sharing, and analysis10.1038/s41597-020-0523-6 ↗
9. Social Media Use, eHealth Literacy, Disease Knowledge, and Preventive Behaviors in the COVID-19 Pandemic: Cross-Sectional Study on Chinese Netizens10.2196/19684 ↗
10. Disease knowledge and self-management behavior of COPD patients in China10.1097/MD.0000000000014460 ↗
11. Romanov, A., and Shivade, C.P. (2018). Lessons from Natural Language Inference in the Clinical Domain. arXiv.10.18653/v1/D18-1187 ↗
12. NCBI disease corpus: A resource for disease name recognition and concept normalization10.1016/j.jbi.2013.12.006 ↗
13. Du, X., Shao, J., and Cardie, C. (2017). Learning to ask: Neural question generation for reading comprehension. arXiv.10.18653/v1/P17-1123 ↗
14. Tang, D., Duan, N., Qin, T., Yan, Z., and Zhou, M. (2017). Question answering and question generation as dual tasks. arXiv.
15. Kim, Y., Lee, H., Shin, J., and Jung, K. (2019). Improving neural question generation using answer separation. In Proceedings of the AAAI Conference on Artificial Intelligence. arXiv.10.1609/aaai.v33i01.33016602 ↗
16. Song, L., Wang, Z., Hamza, W.Z., Zhang, Y., and Gildea, D. (2018, January 1–6). Leveraging context information for natural question generation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA. Available online: https://aclanthology.org/N18-2090.10.18653/v1/N18-2090 ↗
17. Lewis, P., Denoyer, L., and Riedel, S. (2019). Unsupervised question answering by cloze translation. arXiv.10.18653/v1/P19-1484 ↗
18. Chen, Y., Wu, L., and Zaki, M.J. (2019). Reinforcement learning based graph-to-sequence model for natural question generation. arXiv.
19. Language models are unsupervised multitask learners
20. Xue, L., Constant, N., Roberts, A., Kale, M., AI-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gonez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems. arXiv.
22. Jin, Q., Yuan, Z., Xiong, G., and Yu, Q. (2021). Biomedical question answering: A comprehensive review. arXiv.
23. External features enriched model for biomedical question answering10.1186/s12859-021-04176-7 ↗
24. Akdemir, A., and Shibuya, T. (2020). Transfer Learning for Biomedical Question Answering. CLEF (Working Notes), Available online: http://ceur-ws.org/Vol-2696/paper_66.pdf.
25. Jeong, M., Sung, M., Kim, G., Kim, D., Yoon, W., Yoo, J., and Kang, J. (2020). Transferability of natural language inference to biomedical question answering. arXiv.
26. Sarrouti, M., Gupta, D., Abacha, A.B., and Demner-Fushman, D. (2021, November 22). NLM at BioASQ Synergy 2021: Deep Learning-Based Methods for Biomedical Semantic Question Answering about COVID-19. Available online: http://ceur-ws.org/Vol-2936/paper-25.pdf.
27. SemBioNLQA: A semantic biomedical question answering system for retrieving exact and ideal answers to natural language questions10.1016/j.artmed.2019.101767 ↗
28. Gouthaman, K.V., and Mittal, A. (2020, January 23–28). Reducing language biases in visual question answering with visually-grounded question encoder. Proceedings of Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Part XIII 16.s.
29. Peng, K., Yin, C., Rong, W., Lin, C., Zhou, D., and Xiong, Z. (2021). Named Entity Aware Transfer Learning for Biomedical Factoid Question Answering. IEEE/ACM Trans. Comput. Biol. Bioinform.10.1109/TCBB.2021.3079339 ↗
30. Pergola, G., Kochkina, E., Gui, L., Liakata, M., and He, Y. (2021). Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies. arXiv.10.18653/v1/2021.eacl-main.169 ↗
31. Yadav, S., Sarrouti, M., and Gupta, D. (2021). NLM at MEDIQA 2021: Transfer Learning-based Approaches for Consumer Question and Multi-Answer Summarization. Proceedings of the 20th Workshop on Biomedical Language Processing (BIONLP 2021), Association for Computational Linguistics.10.18653/v1/2021.bionlp-1.34 ↗
32. Yue, X., Zhang, X., Yao, Z., Lin, S., and Sun, H. (2020). CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering. arXiv.10.1109/BIBM52615.2021.9669300 ↗
33. Suwarningsih, W. (2021). e-Health Education Using Automatic Question Generation-Based Natural Language (Case Study: Respiratory Tract Infection). Emerging Technologies in Biomedical Engineering and Sustainable TeleMedicine, Springer.10.1007/978-3-030-14647-4_6 ↗
34. Esteva, A., Kale, A., Paulu, S.R., Hashimoto, K., Yin, W., Radev, D., and Socher, R. (2020). Co-search: COVID-19 information retrieval with semantic search, question answering, and abstractive summarization. arXiv.10.1038/s41746-021-00437-0 ↗
35. Papanikolaou, Y., and Pierleoni, A. (2020). DARE: Data Augmented Relation Extraction with GPT-2. arXiv.
36. Oniani, D., and Wang, Y. (2020, January 21–24). A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19. Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. Association for Computing Machinery, Virtual Event.10.1145/3388440.3412413 ↗
37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv.
38. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv.10.18653/v1/W18-5446 ↗
39. Roberts, A., Raffel, C., and Shazeer, N. (2020). How much knowledge can you pack into the parameters of a language model?. arXiv.10.18653/v1/2020.emnlp-main.437 ↗
40. Ngai, H., Park, Y., Chen, J., and Parsa, M. (2021). Transfermer-Based Models for Question Answering on COVID19. arXiv.
41. He, Y., Zhu, Z., Zhang, Y., Chen, Q., and Caverlee, J. (2020). Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition. arXiv.10.18653/v1/2020.emnlp-main.372 ↗
42. Abacha, A.B., Shivade, C., and Demner-Fushman, D. (2019, January 1). Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. Proceedings of the 18th BioNLP Workshop and Shared Task, Florence, Italy. Available online: https://aclanthology.org/W19-5039.pdf.
43. Abacha, A.B., Agichtein, E., Pinter, Y., and Demner-Fushman, D. (2021, November 22). Overview of the Medical Question Answering Task at TREC 2017 LiveQA, Available online: https://trec.nist.gov/pubs/trec26/papers/Overview-QA.pdf.
44. Lee, J.-S., and Hsiang, J. (2020). Patent claim generation by fine-tuning OpenAI GPT-2. arXiv.10.1016/j.wpi.2020.101983 ↗
45. Prismana, I., Prehanto, D.R., Dermawan, D.A., Herlingga, A.C., and Wibawa, S.C. (2021). Nazief & Adriani Stemming Algorithm With Cosine Similarity Method For Integrated Telegram Chatbots With Service. IOP Conference Series: Materials Science and Engineering, IOP. Workshop on Environmental Science, Society, and Technology (WESTECH 2020).10.1088/1757-899X/1125/1/012039 ↗
46. Cer, D., Yang, Y., Kong, S.Y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Céspedes, M., Yuan, S., and Tar, C. (2018). Universal sentence encoder. arXiv.10.18653/v1/D18-2029 ↗