Ontology Attention Layer for Medical Named Entity Recognition
Source abstract
Abstract
Named entity recognition (NER) is particularly challenging for medical texts due to the high domain specificity, abundance of technical terms, and sparsity of data in this field. In this work, we propose a novel attention layer, called the “ontology attention layer”, that enhances the NER performance of a language model for clinical text by utilizing an ontology consisting of conceptual classes related to the target entity set. The proposed layer computes the relevance between each input token and the classes in the ontology and then fuses the encoded token vectors and the class vectors to enhance the token vectors by explicit superior knowledge. In our experiments, we apply the proposed layer to various language models for an NER task based on a Chinese clinical dataset to evaluate the performance of the layer. We also investigate the influence of the granularity of the classes utilized in the ontology attention layer. The experimental results show that the proposed ontology attention layer improved F1 scores by 0.4% to 0.5%. The results suggest that the proposed method is an effective approach to improving the NER performance of existing language models for clinical datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Biomedical Text Mining and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Hubei University of Technology, Wuhan 430068, China
Funding
National College Students Innovation and Entrepreneurship Training Program, China
References (34)
Show source-provided references
1. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016, January 12–17). Neural Architectures for Named Entity Recognition. Proceedings of the North American Chapter of the Association for Computational Linguistics, San Diego, CA, USA.10.18653/v1/N16-1030 ↗
2. Named Entity Recognition of Medical Text Based on the Deep Neural Network
3. Named Entity Recognition in Chinese Electronic Medical Records Based on the Model of Bidirectional Long Short-Term Memory with a Conditional Random Field Layer
4. Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., and Wang, G. (2023). GPT-NER: Named Entity Recognition via Large Language Models. arXiv.
5. Jiang, H., Zhang, D., Cao, T., Yin, B., and Zhao, T. (2021, January 22–27). Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.140 ↗
6. Li, J., Ding, H., Shang, J., McAuley, J., and Feng, Z. (2021). Weakly Supervised Named Entity Tagging with Learnable Logical Rules. arXiv.10.18653/v1/2021.acl-long.352 ↗
7. Li, Y., Shetty, P., Liu, L., Zhang, C., and Song, L. (2021). BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition. arXiv.10.1145/3534678.3539247 ↗
8. Aly, R., Vlachos, A., and McDonald, R. (2021, January 22–27). Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.120 ↗
9. Wu, S., Song, X., and Feng, Z. (2021). MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition. arXiv.10.18653/v1/2021.acl-long.121 ↗
10. Wang, X., Jiang, Y., Bach, N., Wang, T., Huang, Z., Huang, F., and Tu, K. (2021, January 22–27). Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.142 ↗
11. Wei, X., Wang, S., Zhang, D., Bhatia, P., and Arnold, A. (2021). Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey. arXiv.
12. Zhou, B., Cai, X., Zhang, Y., and Yuan, X. (2021, January 22–27). An End-to-End Progressive Multi-Task Learning Framework for Medical Named Entity Recognition and Normalization. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.485 ↗
13. Ji, Z., Xia, T., Han, M., and Xiao, J. (2021, January 22–27). A Neural Transition-based Joint Model for Disease Named Entity Recognition and Normalization. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.219 ↗
14. Wang, Y., Shindo, H., Matsumoto, Y., and Watanabe, T. (2021, January 22–27). Nested Named Entity Recognition via Explicitly Excluding the Influence of the Best Path. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.275 ↗
15. Li, F., Wang, Z., Hui, S.C., Liao, L., Song, D., Xu, J., He, G., and Jia, M. (2021, January 22–27). Modularized Interaction Network for Named Entity Recognition. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.17 ↗
16. Wang, Y., Yu, B., Zhu, H., Liu, T., Yu, N., and Sun, L. (2021, January 22–27). Discontinuous Named Entity Recognition as Maximal Clique Discovery. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2021.acl-long.63 ↗
17. Li, F., Lin, Z., Zhang, M., and Ji, D. (2021). A Span-Based Model for Joint Overlapped and Discontinuous Named Entity Recognition. arXiv.10.18653/v1/2021.acl-long.372 ↗
18. Luong, T., Pham, H., and Manning, C.D. (2015). Effective Approaches to Attention-based Neural Machine Translation. arXiv.10.18653/v1/D15-1166 ↗
19. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E.H. (2016, January 12–17). Hierarchical Attention Networks for Document Classification. Proceedings of the North American Chapter of the Association for Computational Linguistics, San Diego, CA, USA.10.18653/v1/N16-1174 ↗
20. Top-Down Neural Attention by Excitation Backprop10.1007/s11263-017-1059-x ↗
21. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. (2017, January 6–11). Convolutional Sequence to Sequence Learning. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
22. Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the NIPS, Long Beach, CA, USA.
23. An Attention-Based Deep Learning Approach for Sleep Stage Classification With Single-Channel EEG10.1109/TNSRE.2021.3076234 ↗
24. Li, X., Xiong, H., Wang, H., Rao, Y., Liu, L., and Huan, J. (2019). DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks. arXiv.
25. Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
26. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020, January 16–20). Revisiting Pre-Trained Models for Chinese Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, Online.10.18653/v1/2020.findings-emnlp.58 ↗
27. Pre-Training With Whole Word Masking for Chinese BERT10.1109/TASLP.2021.3124365 ↗
28. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
29. A semantic matching energy function for learning with multi-relational data
30. Dettmers, T., Minervini, P., Stenetorp, P., and Riedel, S. (2018, January 2–7). Convolutional 2D Knowledge Graph Embeddings. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11573 ↗
31. Kipf, T.N., and Welling, M. (2017, January 24–26). Semi-Supervised Classification with Graph Convolutional Networks. Proceedings of the International Conference on Learning Representations, Toulon, France.
32. Vashishth, S., Sanyal, S., Nitin, V., and Talukdar, P. (2020, January 26–30). Composition-based Multi-Relational Graph Convolutional Networks. Proceedings of the International Conference on Learning Representations, Online.
33. Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., and Huang, F. (2022, January 22–27). CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.10.18653/v1/2022.acl-long.544 ↗
34. NCBI disease corpus: A resource for disease name recognition and concept normalization10.1016/j.jbi.2013.12.006 ↗