Analysis of the Full-Size Russian Corpus of Internet Drug Reviews with Complex NER Labeling Using Deep Learning Neural Networks and Language Models
Source abstract
Abstract
The paper presents the full-size Russian corpus of Internet users’ reviews on medicines with complex named entity recognition (NER) labeling of pharmaceutically relevant entities. We evaluate the accuracy levels reached on this corpus by a set of advanced deep learning neural networks for extracting mentions of these entities. The corpus markup includes mentions of the following entities: medication (33,005 mentions), adverse drug reaction (1778), disease (17,403), and note (4490). Two of them—medication and disease—include a set of attributes. A part of the corpus has a coreference annotation with 1560 coreference chains in 300 documents. A multi-label model based on a language model and a set of features has been developed for recognizing entities of the presented corpus. We analyze how the choice of different model components affects the entity recognition accuracy. Those components include methods for vector representation of words, types of language models pre-trained for the Russian language, ways of text normalization, and other pre-processing methods. The sufficient size of our corpus allows us to study the effects of particularities of annotation and entity balancing. We compare our corpus to existing ones by the occurrences of entities of different types and show that balancing the corpus by the number of texts with and without adverse drug event (ADR) mentions improves the ADR recognition accuracy with no notable decline in the accuracy of detecting entities of other types. As a result, the state of the art for the pharmacological entity extraction task for the Russian language is established on a full-size labeled corpus. For the ADR entity type, the accuracy achieved is 61.1% by the F1-exact metric, which is on par with the accuracy level for other language corpora with similar characteristics and ADR representativeness. The accuracy of the coreference relation extraction evaluated on our corpus is 71%, which is higher than the results achieved on the other Russian-language corpora.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Biomedical Text Mining and Ontologies · Topic Modeling · Misinformation and Its Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Complex of NBICS Technologies, National Research Centre “Kurchatov Institute”, 123182 Moscow, Russia
Moscow Engineering Physics Institute, National Research Nuclear University, 115409 Moscow, Russia
National Center for Cognitive Research, ITMO University, 197101 Saint Petersburg, Russia
Department of NBIC-Technologies, Moscow Institute of Physics and Technology, 141701 Dolgoprudny, Russia
Funding
Russian Science Foundation
References (69)
Show source-provided references
1. Are Artificial Intelligence (AI) And Machine Learning (ML) Having An Effective Role In Helping Humanity Address The New Coronavirus Pandemic?10.37394/23208.2020.17.14 ↗
2. An Artificial Intelligence Approach Based on Hybrid CNN-XGB Model to Achieve High Prediction Accuracy through Feature Extraction, Classification and Regression for Enhancing Drug Discovery in Biomedicine10.46300/91011.2021.15.22 ↗
3. Cadec: A corpus of adverse drug event annotations10.1016/j.jbi.2015.03.010 ↗
4. TwiMed: Twitter and PubMed comparable corpus of drugs, diseases, symptoms, and their relations10.2196/publichealth.6396 ↗
5. Sarker, A., Nikfarjam, A., and Gonzalez, G. (2016). Social media mining shared task workshop. Biocomputing 2016: Proceedings of the Pacific Symposium, World Scientific.10.1142/9789814749411_0054 ↗
6. Portable automatic text classification for adverse drug reaction detection via multi-corpus training10.1016/j.jbi.2014.11.002 ↗
7. The PsyTAR dataset: From patients generated narratives to a corpus of adverse drug events and effectiveness of psychiatric medications10.1016/j.dib.2019.103838 ↗
8. Basaldella, M., and Collier, N. (2019, January 3). BioReddit: Word embeddings for user-generated biomedical NLP. Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), Hong Kong, China.10.18653/v1/D19-6205 ↗
9. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records10.1093/jamia/ocz166 ↗
10. The Russian Drug Reaction Corpus and neural models for drug reactions and effectiveness detection in user reviews10.1093/bioinformatics/btaa675 ↗
11. NEHTA (2014). Australian Medicines Terminology v3 Model–Common v1.4, National E-Health Transition Authority. Tech. rep. EP-1825:2014.
12. The SIDER database of drugs and side effects10.1093/nar/gkv1075 ↗
13. Gupta, S., Gupta, M., Varma, V., Pawar, S., Ramrakhiyani, N., and Palshikar, G.K. (2018). Co-training for extraction of adverse drug reaction mentions from tweets. European Conference on Information Retrieval, Springer.10.1007/978-3-319-76941-7_44 ↗
14. Lexicon Knowledge Boosted Interaction Graph Network for Adverse Drug Reaction Recognition from Social Media10.1109/JBHI.2020.3042549 ↗
15. Wang, W. (January, January USA). Mining adverse drug reaction mentions in twitter with word embeddings. Proceedings of the Social Media Mining Shared Task Workshop at the Pacific Symposium on Biocomputing, Kohala Coast, HI.
16. Gupta, S., Gupta, M., Varma, V., Pawar, S., Ramrakhiyani, N., and Palshikar, G.K. (2018). Multi-Task Learning for Extraction of Adverse Drug Reaction Mentions from Tweets. European Conference on Information Retrieval, Springer.10.1007/978-3-319-76941-7_5 ↗
17. Exploiting adversarial transfer learning for adverse drug reaction detection from texts10.1016/j.jbi.2020.103431 ↗
18. Sankaran, N., Kaivalya, M., Sreeranga, R., and Venkat, R. (2020, January 19). Evaluation of Transfer Learning for Adverse Drug Event (ADE) and Medication Entity Extraction. Proceedings of the 3rd Clinical Natural Language Processing Workshop, Online.
19. Named entity recognition with bidirectional LSTM-CNNs10.1162/tacl_a_00104 ↗
20. Magge, A., Klein, A., Miranda-Escalada, A., Al-Garadi, M.A., Alimova, I., Miftahutdinov, Z., Farre, E., Lima-López, S., Flores, I., and O’Connor, K. (2021, January 10). Overview of the sixth social media mining for health applications (# smm4h) shared tasks at NAACL 2021. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.4 ↗
21. Zhou, T., Li, Z., Gan, Z., Zhang, B., Chen, Y., Niu, K., Wan, J., Liu, K., Zhao, J., and Shi, Y. (2021, January 10). Classification, extraction, and normalization: Casia_unisound team at the social media mining for health 2021 shared tasks. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.13 ↗
22. Sakhovskiy, A., Miftahutdinov, Z., and Tutubalina, E. (2021, January 10). KFU NLP Team at SMM4H 2021 Tasks: Cross-lingual and Cross-modal BERT-based Models for Adverse Drug Effects. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.6 ↗
23. El-karef, M., and Hassan, L. (2021, January 10). A Joint Training Approach to Tweet Classification and Adverse Effect Extraction and Normalization for SMM4H 2021. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.16 ↗
24. Dima, G.A., Cercel, D.C., and Dascalu, M. (2021, January 10). Transformer-based Multi-Task Learning for Adverse Effect Mention Analysis in Tweets. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.7 ↗
25. Ji, Z., Xia, T., and Han, M. (2021, January 10). PAII-NLP at SMM4H 2021: Joint Extraction and Normalization of Adverse Drug Effect Mentions in Tweets. Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task, Online.10.18653/v1/2021.smm4h-1.26 ↗
26. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
27. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
28. On the accuracy of different neural language model approaches to ADE extraction in natural language corpora10.1016/j.procs.2021.06.082 ↗
29. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012, January 13). CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. Proceedings of the Joint Conference on EMNLP and CoNLL-Shared Task, Jeju Island, Korea.
30. Mind the GAP: A Balanced Corpus of Gendered Ambiguou10.1162/tacl_a_00240 ↗
31. Annotation and detection of drug effects in text for pharmacovigilance10.1186/s13321-018-0290-y ↗
32. Toldova, S., Roytberg, A., Ladygina, A.A., Vasilyeva, M.D., Azerkovich, I.L., Kurzukov, M., Sim, G., Gorshkov, D.V., Ivanova, A., and Nedoluzhko, A. (2014, January 4–8). RU-EVAL-2014: Evaluating Anaphora and Coreference Resolution for Russian. Proceedings of the International Conference on Computational Linguistics and Intellectual Technologies “Dialogue”, Bekasovo, Russia.
33. Ju, T.S. (June, January 29). RU-EVAL-2019: Evaluating Anaphora And Coreference Resolution For Russian. Proceedings of the International Conference on Computational Linguistics and Intellectual Technologies “Dialogue”, Moscow, Russia.
34. Lee, K., He, L., Lewis, M., and Zettlemoyer, L. (2017). End-to-end neural coreference resolution. arXiv.10.18653/v1/D17-1018 ↗
35. Lee, K., He, L., and Zettlemoyer, L. (2018). Higher-order coreference resolution with coarse-to-fine inference. arXiv.10.18653/v1/N18-2108 ↗
36. Joshi, M., Levy, O., Weld, D.S., and Zettlemoyer, L. (2019). BERT for coreference resolution: Baselines and analysis. arXiv.10.18653/v1/D19-1588 ↗
37. Xu, L., and Choi, J.D. (2020). Revealing the myth of higher-order inference in coreference resolution. arXiv.10.18653/v1/2020.emnlp-main.686 ↗
38. Spanbert: Improving pre-training by representing and predicting spans10.1162/tacl_a_00300 ↗
39. Toshniwal, S., Wiseman, S., Ettinger, A., Livescu, K., and Gimpel, K. (2020). Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks. arXiv.10.18653/v1/2020.emnlp-main.685 ↗
40. Rosminzdrav (2021, December 12). State Register of Registered Drugs in Russia. Available online: http://grls.rosminzdrav.ru/.
41. World Health Organization (2021, December 12). International Statistical Classification of Diseases. Available online: https://icd.who.int/browse10/2019/en.
42. A new drug classification for computer systems: The ATC extension code10.1016/0020-7101(95)01135-2 ↗
43. Pattern-matching-the gestalt approach
44. Straka, M., Hajic, J., and Strakov’a, J. (2016, January 23–28). UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia.
45. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
46. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv.10.18653/v1/2020.acl-main.747 ↗
47. Suero Montero, C., Munezero, M., and Kakkonen, T. (2014, January 6–12). Investigating the role of emotion-based features in author gender classification of text. Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics, Kathmandu, Nepal.10.1007/978-3-642-54903-8_9 ↗
48. The psychological meaning of words: LIWC and computerized text analysis methods10.1177/0261927X09351676 ↗
49. Litvinova, O., Seredin, P., Litvinova, T., and Lyell, J. (2017, January 3–7). Deception detection in Russian texts. Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain.10.18653/v1/E17-4005 ↗
50. A quantitative method of text emotiveness evaluation on base of the psycholinguistic markers founded on morphological features10.1016/j.procs.2015.11.036 ↗
51. Tolmachova, E. (2021, December 12). VIDAL: Directory of medicines in Russia; Vidal RusVIDAL: Directory of medicines in Russia. Available online: https://www.vidal.ru/.
52. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
53. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
54. Schuster, M., and Nakajima, K. (2012, January 25–30). Japanese and Korean voice search. Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6289079 ↗
55. Kudo, T., and Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv.10.18653/v1/D18-2012 ↗
56. Imagenet classification with deep convolutional neural networks
57. A theoretically grounded application of dropout in recurrent neural networks
58. Dumoulin, V., and Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv.
59. Boureau, Y.L., Ponce, J., and LeCun, Y. (2010, January 21–24). A theoretical analysis of feature pooling in visual recognition. Proceedings of the 27th International conference on machine learning (ICML-10), Haifa, Israel.
60. Lafferty, J.D., McCallum, A., and Pereira, F.C.N. (July, January 28). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, Williamstown, MA, USA.
61. Rajapakse, T.C. (2021, December 12). Simple Transformers. Available online: https://github.com/ThilinaRajapakse/simpletransformers.
62. Biewald, L. (2021, December 12). Experiment Tracking with Weights and Biases. Available online: wandb.com.
63. Moosavi, N.S., and Strube, M. (2016, January 7–12). Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1060 ↗
64. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018, January 7–12). Learning Word Vectors for 157 Languages. Proceedings of the International Conference on Language Resources and Evaluation, Miyazaki, Japan.
65. Burtsev, M., Seliverstov, A., Airapetyan, R., Arkhipov, M., Baymurzina, D., Bushkov, N., Gureenkova, O., Khakhulin, T., Kuratov, Y., and Kuznetsov, D. (2018, January 15–20). DeepPavlov: Open-source library for dialogue systems. Proceedings of the ACL 2018, System Demonstrations, Melbourne, Australia.10.18653/v1/P18-4021 ↗
66. Koehn, P. (2021, December 12). Statmt—Internet Resource about Research in the Field of Statistical Machine Translation. Available online: https://www.statmt.org/.
67. Smith, L.N. (2017, January 24–31). Cyclical learning rates for training neural networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.58 ↗
68. System Center Service Manager Library (2021, December 12). Russian Version of the Medical Subject Headings (MeSH) Database, Available online: https://www.nlm.nih.gov/mesh/meshhome.html.
69. Shelmanov, A., Smirnov, I., and Vishneva, E. (2015, January 27–30). Information extraction from clinical texts in Russian. Proceedings of the International Conference on Computer Linguistics and Intellectual Technologies “Dialogue”, Moscow, Russia.