Survey of BERT-Base Models for Scientific Text Classification: COVID-19 Case Study
Source abstract
Abstract
On 30 January 2020, the World Health Organization announced a new coronavirus, which later turned out to be very dangerous. Since that date, COVID-19 has spread to become a pandemic that has now affected practically all regions in the world. Since then, many researchers in medicine have contributed to fighting COVID-19. In this context and given the great growth of scientific publications related to this global pandemic, manual text and data retrieval has become a challenging task. To remedy this challenge, we are proposing CovBERT, a pre-trained language model based on the BERT model to automate the literature review process. CovBERT relies on prior training on a large corpus of scientific publications in the biomedical domain and related to COVID-19 to increase its performance on the literature review task. We evaluate CovBERT on the classification of short text based on our scientific dataset of biomedical articles on COVID-19 entitled COV-Dat-20. We demonstrate statistically significant improvements by using BERT.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Misinformation and Its Impacts · COVID-19 diagnosis using AI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Engineering School of Sfax (ENIS), University of Sfax, Sfax 3038, Tunisia
ReDCAD Laboratory, Department of Computer Engineering and Applied Mathematics, University of Sfax, Sfax 3029, Tunisia
Solutions Galore Inc., Moncton, NB E1C 5Y1, Canada
Faculty of Engineering, Université de Moncton, Moncton, NB E1A 3E9, Canada
International Institute of Technology and Management, Commune d’Akanda, Libreville BP 1989, Gabon
Spectrum of Knowledge Production & Skills Development, Sfax 3027, Tunisia
Department of Electrical and Electronic Engineering Science, School of Electrical Engineering, University of Johannesburg, Johannesburg 2006, South Africa
References (40)
Show source-provided references
1. Coronavirus disease 2019 (COVID-19): A perspective from china10.1148/radiol.2020200490 ↗
2. (2020, January 30). Worldometers for COVID-19. Available online: https://www.worldometers.info/.
3. Pan, S., and Yang, Q. (2010). A survey on transfer learning. Knowledge and Data Engineering. IEEE Trans., 1345–1359.10.1109/TKDE.2009.191 ↗
4. Yousaf, A., Asif, R.M., Shakir, M., Rehman, A.U., Alassery, F., Hamam, H., and Cheikhrouhou, O. (2021). A Novel Machine Learning-Based Price Forecasting for Energy Management Systems. Sustainability, 13.10.3390/su132212693 ↗
5. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
6. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
7. Alt, C., Hübner, M., and Hennig, L. (August, January 28). Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
8. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
9. A recent overview of the state-of-the-art elements of text classification10.1016/j.eswa.2018.03.058 ↗
10. Holzinger, A., Kieseberg, P., Weippl, E., and Tjoa, A.M. (2018, January 27–30). Current Advances, Trends and Challenges of Machine Learning and Knowledge Extraction: From Machine Learning to Explainable AI. Proceedings of the International Cross-Domain Conference CD-MAKE 2018, Hamburg, Germany.10.1007/978-3-319-99740-7 ↗
11. Maas, A., Daly, R., Pham, P., Huang, D., Ng, A., and Potts, C. (2011, January 19–24). Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
12. Zhang, D. (August, January 28). Question classification using support vector machines. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, ON, Canada.
13. Wang, S., and Manning, C. (2012, January 8–14). Baselines and bigrams: Simple, good sentiment and topic classification. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers, Jeju Island, Korea.
14. A New V-Net Convolutional Neural Network Based on Four-Dimensional Hyperchaotic System for Medical Image Encryption
15. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modelling. IPS 2014 Workshop on Deep Learning. arXiv.
16. Artificial neural networks in medical diagnosis
17. Shi, M., Wang, K., and Li, C. (2019, January 17–19). A C-LSTM with Word Embedding Model for News Text Classification. Proceedings of the IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS), Beijing, China.10.1109/ICIS46139.2019.8940289 ↗
18. Xiao, Y., and Cho, K. (2016). Efficient character-level document classification by combining convolution and recurrent layers. arXiv.
19. Wang, B. (2018, January 15–20). Disconnected recurrent neural networks for text categorization. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1215 ↗
20. Application of artificial neural network in tunnel engineering: A systematic review10.1109/ACCESS.2020.3004995 ↗
21. Zheng, S., and Yang, M. (2019, January 17–20). A New Method of Improving BERT for Text Classification. Proceedings of the Intelligence Science and Big Data Engineering. Big Data and Machine Learning, Nanjing, China.10.1007/978-3-030-36204-1_37 ↗
22. Yao, L., Zhang, Y., Wei, B., Li, Z., and Huang, X. (2016, January 15–18). Traditional chinese medicine clinical records classification using knowledge-powered document embedding. Proceedings of the 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China.
23. Active learning for clinical text classification: Is it better than random sampling?10.1136/amiajnl-2011-000648 ↗
24. Knowledge-based biomedical word sense disambiguation: An evaluation and application to clinical document classification10.1136/amiajnl-2012-001350 ↗
25. Clinical text classification with rule-based features and knowledge-guided convolutional neural networks
26. Asgari-Chenaghlu, M. (2017). Word Vector Representation, Word2vec, Glove, and Many More Explained. [Ph.D. Thesis, University of Tabriz].
27. Howard, J., and Ruder, S. (2018, January 15–20). Universal language model fine-tuning for text classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1031 ↗
28. Radford, A., Karthik, N., Tim, S., and Ilya, S. (2021, October 01). Improving Language Understanding by Generative Pre-Training. Available online: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
29. Term-weighting approaches in automatic text retrieval10.1016/0306-4573(88)90021-0 ↗
30. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017, January 3–7). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain.10.18653/v1/E17-2068 ↗
31. A systematic analysis of performance measures for classification tasks10.1016/j.ipm.2009.03.002 ↗
32. Bird, S. (2006, January 17–18). NLTK: The natural language toolkit. Proceedings of the COLING/ACL on Interactive Presentation Sessions Association for Computational Linguistics 2006, Sydney, Australia.10.3115/1225403.1225421 ↗
33. A review of recurrent neural networks: Lstm cells and network architectures10.1162/neco_a_01199 ↗
34. Xu, J., and Du, Q. (2019, January 10–12). A deep investigation into fasttext. Proceedings of the IEEE 21st International Conference on High Performance Computing and Communications, Zhangjiajie, China.10.1109/HPCC/SmartCity/DSS.2019.00234 ↗
35. Biobert: A pre-trained biomedical language representation model for biomedical text mining10.1093/bioinformatics/btz682 ↗
36. Huang, K., Altosaar, J., and Ranganath, R. (2019). Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv.
37. Beltagy, I., Cohan, A., and Lo, K. (2019, January 3–7). Scibert: Pretrained contextualized embeddings for scientific text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019, HongKong, China.
38. Matthew, E.P., Mark, N., Robert, L., Roy, S., Vidur, J., Sameer, S., and Noah, A.S. (2019, January 3–7). Knowledge Enhanced Contextual Word Representations. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), HongKong, China.
39. Ashish, V., Noam, S., Niki, P., Jakob, U., Llion, J., Aidan, N.G., Lukasz, K., and Illia, P. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
40. Khadhraoui, M., Bellaaj, H., Ben Ammar, M., Hamam, H., and Jmaiel, M. (2020, January 24–26). Machine Learning Classification Models with SPD/ED Dataset: Comparative Study of Abstract Versus Full Article Approach. Proceedings of the ICOST 2020, Hammamet, Tunisia.10.1007/978-3-030-51517-1_31 ↗