MSA K-BERT: A Method for Medical Text Intent Classification
Source abstract
Abstract
Improving medical text intent classification accuracy can assist the medical field in achieving more precise diagnoses. However, existing methods suffer from problems such as low accuracy and a lack of knowledge supplementation. To address these challenges, this paper proposes MSA K-BERT, a knowledge-enhanced bidirectional encoder representation model that integrates a multi-scale attention (MSA) mechanism to enhance prediction performance while solving critical issues like heterogeneity of embedding spaces and knowledge noise. We systematically validate the reliability of this model on medical text intent classification datasets and compare it with various deep learning models. The research results indicate that MSA K-BERT makes the following key contributions: First, it introduces a knowledge-supported language representation model compatible with BERT, enhancing language representations through the refined injection of knowledge graphs. Second, it adopts a multi-scale attention mechanism to reinforce different feature layers, significantly improving the model’s accuracy and interpretability. Especially in the IMCS-21 dataset, MSA K-BERT achieves precision, recall, and F1 scores of 0.826, 0.794, and 0.810, respectively, all exceeding the current mainstream methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Text and Document Classification Technologies · Topic Modeling · Biomedical Text Mining and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Biomedical and Pharmaceutical Sciences, Guangdong University of Technology, 100 Waihuanxi Road, Guangzhou 510006, China
References (63)
Show source-provided references
1. Chen, N., Su, X., Liu, T., Hao, Q., and Wei, M. (2020). A Benchmark Dataset and Case Study for Chinese Medical Question Intent Classification. BMC Med. Inf. Decis. Mak., 20.10.1186/s12911-020-1122-3 ↗
2. Medical Text Classification Based on Convolutional Neural Network: A Review
3. Yao, L., Mao, C., and Luo, Y. (2019). Clinical Text Classification with Rule-Based Features and Knowledge-Guided Convolutional Neural Networks. BMC Med. Inf. Decis. Mak., 19.10.1186/s12911-019-0781-4 ↗
4. Mollaei, N., Cepeda, C., Rodrigues, J., and Gamboa, H. (2022, January 5). Biomedical Text Mining: Applicability of Machine Learning-Based Natural Language Processing in Medical Database. Proceedings of the Biosignals, Caparica, Portugal.10.5220/0010819500003123 ↗
5. Lenivtceva, I., Slasten, E., Kashina, M., and Kopanitsa, G. (2020). Applicability of Machine Learning Methods to Multi-Label Medical Text Classification. Lecture Notes in Computer Science, Springer International Publishing.10.1007/978-3-030-50423-6_38 ↗
6. Analytics of Machine Learning-Based Algorithms for Text Classification10.1016/j.susoc.2022.03.001 ↗
7. Deep Learning—Based Text Classification: A Comprehensive Review10.1145/3439726 ↗
8. Shen, Z., and Zhang, S. (2020, January 30). A Novel Deep-Learning-Based Model for Medical Text Classification. Proceedings of the 2020 9th International Conference on Computing and Pattern Recognition, Xiamen, China.10.1145/3436369.3436469 ↗
9. A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU10.32604/jai.2024.054314 ↗
10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 1). Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MI, USA.
11. Luan, Y., and Lin, S. (2019, January 29–31). Research on Text Classification Based on CNN and LSTM. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), Dalian, China.10.1109/ICAICA.2019.8873454 ↗
12. Kim, Y. (2014, January 25–29). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1181 ↗
13. Text-Attentional Convolutional Neural Network for Scene Text Detection10.1109/TIP.2016.2547588 ↗
14. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014). A Convolutional Neural Network for Modelling Sentences. arXiv.10.3115/v1/P14-1062 ↗
15. Zhang, J., Li, Y., Tian, J., and Li, T. (2018, January 12–14). LSTM-CNN Hybrid Model for Text Classification. Proceedings of the 2018 IEEE 3rd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China.10.1109/IAEAC.2018.8577620 ↗
16. Attention Mechanisms in Computer Vision: A Survey10.1007/s41095-022-0271-y ↗
17. CNN-RNN Based Intelligent Recommendation for Online Medical Pre-Diagnosis Support10.1109/TCBB.2020.2994780 ↗
18. Liu, Z., Huang, H., Lu, C., and Lyu, S. (2020). Multichannel CNN with Attention for Text Classification. arXiv.
19. Dernoncourt, F., and Lee, J.Y. (2017). PubMed 200k RCT: A Dataset for Sequential Sentence Classification in Medical Abstracts. arXiv.10.18653/v1/E17-2110 ↗
20. Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M.R., Weissenborn, D., Krithara, A., Petridis, S., and Polychronopoulos, D. (2015). An Overview of the BIOASQ Large-Scale Biomedical Semantic Indexing and Question Answering Competition. BMC Bioinform., 16.10.1186/s12859-015-0564-6 ↗
21. A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification10.1155/2022/3498123 ↗
22. Zhu, R., Tu, X., and Huang, J.X. (2021). Utilizing BERT for Biomedical and Clinical Text Mining. Data Analytics in Biomedical Engineering and Healthcare, Elsevier.10.1016/B978-0-12-819314-3.00005-7 ↗
23. BERT Applications in Natural Language Processing: A Review10.1007/s10462-025-11162-5 ↗
24. Sentiment Analysis Classification System Using Hybrid BERT Models10.1186/s40537-023-00781-w ↗
25. BERT-CNN: Improving BERT for Requirements Classification Using CNN10.1016/j.procs.2023.01.234 ↗
26. Li, W., Gao, S., Zhou, H., Huang, Z., Zhang, K., and Li, W. (2019, January 4–6). The Automatic Text Classification Method Based on Bert and Feature Union. Proceedings of the 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), Tianjin, China.10.1109/ICPADS47876.2019.00114 ↗
27. A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources
28. Naseem, U., Thapa, S., Zhang, Q., Hu, L., Masood, A., and Nasim, M. (2023, January 14). Reducing Knowledge Noise for Improved Semantic Analysis in Biomedical Natural Language Processing Applications. Proceedings of the 5th Clinical Natural Language Processing Workshop, Toronto, ON, Canada.10.18653/v1/2023.clinicalnlp-1.32 ↗
29. Large Language Models in Medicine10.1038/s41591-023-02448-8 ↗
30. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2024). GPT-4 Technical Report. arXiv.
31. Mullick, A., Gupta, M., and Goyal, P. (2024). Intent Detection and Entity Extraction from BioMedical Literature. arXiv.
32. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
33. Cheng, Y., Zhao, H., Zhou, X., Zhao, J., Cao, Y., Yang, C., and Cai, X. (2025). A Large Language Model for Advanced Power Dispatch. Sci. Rep., 15.10.1038/s41598-025-91940-x ↗
34. Wu, J., Zhou, S., Zuo, S., Chen, Y., Sun, W., Luo, J., Duan, J., Wang, H., and Wang, D. (2021). U-Net Combined with Multi-Scale Attention Mechanism for Liver Segmentation in CT Images. BMC Med. Inf. Decis. Mak., 21.10.1186/s12911-021-01649-w ↗
35. Knowledge Graphs10.1145/3447772 ↗
36. Sengupta, S., Krone, J., and Mansour, S. (2021). On the Robustness of Intent Classification and Slot Labeling in Goal-Oriented Dialog Systems to Real-World Noise. arXiv.10.18653/v1/2021.nlp4convai-1.7 ↗
37. Kalinowski, A., and An, Y. (2020). A Survey of Embedding Space Alignment Methods for Language and Knowledge Graphs. arXiv.
38. Khalid, B., Dai, S., Taghavi, T., and Lee, S. (2024, January 15). Label Supervised Contrastive Learning for Imbalanced Text Classification in Euclidean and Hyperbolic Embedding Spaces. Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024), San Ġiljan, Malta.
39. Text Classification: A Least Square Support Vector Machine Approach10.1016/j.asoc.2006.04.002 ↗
40. Text Document Preprocessing with the Bayes Formula for Classification Using the Support Vector Machine10.1109/TKDE.2008.76 ↗
41. ForesTexter: An Efficient Random Forest Algorithm for Imbalanced Text Categorization10.1016/j.knosys.2014.06.004 ↗
42. Machine Learning Methods for Small Data Challenges in Molecular Science10.1021/acs.chemrev.3c00189 ↗
43. Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions10.1007/s42979-021-00815-1 ↗
44. Bidirectional LSTM with Attention Mechanism and Convolutional Layer for Text Classification10.1016/j.neucom.2019.01.078 ↗
45. Tree-Structured Regional CNN-LSTM Model for Dimensional Sentiment Analysis10.1109/TASLP.2019.2959251 ↗
46. Han, Y., Guo, J., Yang, H., Guan, R., and Zhang, T. (2024). SSMA-YOLO: A Lightweight YOLO Model with Enhanced Feature Extraction and Fusion Capabilities for Drone-Aerial Ship Image Detection. Drones, 8.10.3390/drones8040145 ↗
47. Liu, X., Wang, Z., Han, Y., Wang, Y., Yuan, J., Song, J., Zheng, B., Zhang, L., Huang, S., and Chen, H. (2025). Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models. arXiv.
48. Han, Y., Duan, B., Guan, R., Yang, G., and Zhen, Z. (2024). LUFFD-YOLO: A Lightweight Model for UAV Remote Sensing Forest Fire Detection Based on Attention Mechanism and Multi-Level Feature Fusion. Remote Sens., 16.10.3390/rs16122177 ↗
49. Liu, X., Liu, T., Huang, S., Xin, Y., Hu, Y., Qin, L., Wang, D., Wu, Y., and Chen, H. (2025). M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension. IEEE Transactions on Circuits and Systems for Video Technology. arXiv.10.1109/TCSVT.2025.3551766 ↗
50. Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., and Wu, H. (2019). ERNIE: Enhanced Representation through Knowledge Integration. arXiv.
51. Clark, K., Luong, M.-T., Le, Q.V., and Manning, C.D. (2020). ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators. arXiv.
52. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations. arXiv.
53. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv.10.18653/v1/D16-1264 ↗
54. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv.10.18653/v1/W18-5446 ↗
55. Moradi, M., Blagec, K., and Samwald, M. (2021). Deep Learning Models Are Not Robust against Noise in Clinical Text. arXiv.
56. Enhancing Clinical Concept Extraction with Contextual Embeddings10.1093/jamia/ocz096 ↗
57. Zhang, H., Lu, A.X., Abdalla, M., McDermott, M., and Ghassemi, M. (2020, January 2). Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings. Proceedings of the ACM Conference on Health, Inference, and Learning, Toronto, ON, Canada.10.1145/3368555.3384448 ↗
58. K-Bert: Enabling Language Representation with Knowledge Graph
59. Attention Is All You Need
60. Multi-Scale Attention Convolutional Neural Network for Time Series Classification10.1016/j.neunet.2021.01.001 ↗
61. Chen, W., Li, Z., Fang, H., Yao, Q., Zhong, C., Hao, J., Zhang, Q., Huang, X., Peng, J., and Wei, Z. (2023). A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets. Bioinformatics, 39.10.1093/bioinformatics/btac817 ↗
62. Smart Healthcare: Making Medical Care More Intelligent10.1016/j.glohj.2019.07.001 ↗
63. Effect of Clinical Decision-Support Systems: A Systematic Review10.7326/0003-4819-157-1-201207030-00450 ↗