Multilabel Classification of Radiology Image Concepts Using Deep Learning
Source abstract
Abstract
Understanding and interpreting medical images, particularly radiology images, is a time-consuming task that requires specialized expertise. In this study, we developed a deep learning-based system capable of automatically assigning multiple standardized medical concepts to radiology images, leveraging deep learning models. These concepts are based on Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs) and describe the radiology images in detail. Each image is associated with multiple concepts, making it a multilabel classification problem. We implemented several deep learning models, including DenseNet121, ResNet101, and VGG19, and evaluated them on the ImageCLEF 2020 Medical Concept Detection dataset. This dataset consists of radiology images with multiple CUIs associated with each image and is organized into seven categories based on their modality information. In this study, transfer learning techniques were applied, with the models initially pre-trained on the ImageNet dataset and subsequently fine-tuned on the ImageCLEF dataset. We present the evaluation results based on the F1-score metric, demonstrating the effectiveness of our approach. Our best-performing model, DenseNet121, achieved an F1-score of 0.89 on the classification of the twenty most frequent medical concepts, indicating a significant improvement over baseline methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Radiomics and Machine Learning in Medical Imaging · COVID-19 diagnosis using AI · AI in cancer detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Agricultural Sciences, Food, Natural Resources, and Engineering, University of Foggia, Via Napoli 25, 71121 Foggia, Italy
References (41)
Show source-provided references
1. A Comparative Study of Medical Imaging Techniques
2. Kougia, V. (2019). Medical Image Labeling and Report Generation. [Master’s Thesis, Athens University of Economics and Business (AUEB)].
3. Karatzas, B., Pavlopoulos, J., Kougia, V., and Androutsopoulos, I. (2020, January 22–25). AUEB NLP Group at ImageCLEFmed Caption 2020. Proceedings of the CLEF 2020, Thessaloniki, Greece.
4. Kougia, V., and Androutsopoulos, I. (2019, January 6). A survey on biomedical image captioning. Proceedings of the Second Workshop on Shortcomings in Vision and Language, Minneapolis, MN, USA.
5. Dermatologist-Level Classification of Skin Cancer with Deep Neural Networks10.1038/nature21056 ↗
6. Adopting Semantic Information of Grayscale Radiographs for Image Classification and Retrieval
7. Udas, N., Beuth, F., and Kowerko, D. (2020, January 22–25). Concept Detection in Medical Images using Xception Models-TUCMC at ImageCLEFmed 2020. Proceedings of the ImageCLEF at the CLEF 2020 Conference-Conference and Labs of the Evaluation Forum, Thessaloniki, Greece.
8. Assisted Annotation of Surgical Vide-os Using Deep Learning10.1007/s11548-019-02108-8 ↗
9. Preparing a collection of radiology examinations for distribution and retrieval10.1093/jamia/ocv080 ↗
10. Jing, B., Xie, P., and Xing, E. (2018). On the Automatic Generation of Medical Imaging Reports. arXiv.10.18653/v1/P18-1240 ↗
11. Yuan, J., Liao, H., Luo, R., and Luo, J. (2019, January 13–17). Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32226-7_80 ↗
12. Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (2019). Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv.10.1609/aaai.v33i01.3301590 ↗
13. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (2016, January 27–30). Image captioning with semantic attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
14. Li, C.Y., Liang, X., Hu, Z., and Xing, E.P. (February, January 27). Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), Honolulu, HI, USA.
15. Deep Learning for Ultrasound Image Caption Generation based on Object Detection10.1016/j.neucom.2018.11.114 ↗
16. Cross-Language Evaluation Forum: Objectives, Results, Achievements10.1023/B:INRT.0000009438.69013.fa ↗
17. Ionescu, B., Müller, H., Péteri, R., Abacha, A.B., Datla, V., Hasan, S.A., Demner-Fushman, D., Kozlovski, S., Liauchuk, V., and Cid, Y.D. (2020, January 22–25). Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications. Proceedings of the 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece.10.1007/978-3-030-58219-7_22 ↗
18. Eickhoff, C., Schwall, I., De Herrera, A.G.S., and Müller, H. (2017, January 13–15). Overview of ImageCLEFcaption 2017–image caption prediction and concept detection for biomedical images. Proceedings of the CEUR Workshop Proceedings, Dublin, Ireland.
19. Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T.R., Datla, V., Lee, K., Qadir, A., and Swisher, C. (2017, January 11–14). PRNA at ImageCLEF 2017 Caption Prediction and Concept Detection Tasks. Proceedings of the CLEF (Working Notes), Dublin, Ireland.
20. Abacha, A.B., De Herrera, A.G.S., Gayen, S., Demner-Fushman, D., and Antani, S. (2017, January 13–15). NLM at Im-ageCLEF 2017 caption task. Proceedings of the CEUR Workshop Proceedings, Dublin, Ireland.
21. Valavanis, L., and Stathopoulos, S. (2017, January 11–14). IPL at ImageCLEF 2017 Concept Detection Task. Proceedings of the CLEF (Working Notes), Dublin, Ireland.
22. Garcia Seco De Herrera, A., Eickhof, C., Andrearczyk, V., and Müller, H. (2018, January 10–14). Overview of the ImageCLEF 2018 caption prediction tasks. Proceedings of the CEUR Workshop Proceedings, Avignon, France.
23. Johnson, J., Douze, M., and Jégou, H. (2017). Billion-scale similarity search with GPUs. arXiv.
24. Generative Adver-sarial Nets10.1145/3422622 ↗
25. Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv.
26. Guo, Z., Wang, X., Zhang, Y., and Li, J. (2019, January 9–12). ImageSem at ImageCLEFmed Caption 2019 Task: A Two-stage Medical Concept Detection Strategy. Proceedings of the CLEF (Working Notes), Lugano, Switzerland.
27. Singh, S., Karimi, S., Ho-Shon, K., Hamey, L., Cappellato, L., Ferro, N., Losada, D.E., and Müller, H.-N. (2019, January 9–12). Biomedical Concept Detection in Medical Images: MQ-CSIRO at 2019 ImageCLEFmed Caption Task. Proceedings of the CLEF (Working Notes), Lugano, Switzerland.
28. Kougia, V., Pavlopoulos, J., and Androutsopoulos, I. (2020). Medical Image Tagging by Deep Learning and Retrieval. International Conference of the Cross-Language Evaluation Forum for European Languages, Springer.10.1007/978-3-030-58219-7_14 ↗
29. Xu, J., Liu, W., Liu, C., Wang, Y., Chi, Y., Xie, X., and Hua, X.-S. (2019, January 9–12). Concept Detection based on Multilabel Classification and Image Captioning Approach-DAMO at ImageCLEF 2019. Proceedings of the CLEF (Working Notes), Lugano, Switzerland.
30. Pelka, O., Koitka, S., Rückert, J., Nensa, F., and Friedrich, C.M. (2018). Radiology Objects in COntext (ROCO): A multimodal image dataset. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, Springer.10.1007/978-3-030-01364-6_20 ↗
31. Kalimuthu, M., Nunnari, F., and Sonntag, D. (2020). A competitive deep neural network approach for the im-ageclefmed caption 2020 task. arXiv.
32. ImageCLEF 2020: Image Caption Prediction using Multilabel Convolutional Neural Network
33. Bello, A., Ng, S.-C., and Leung, M.-F. (2024). Skin Cancer Classification Using Fine-Tuned Transfer Learning of DENSENET-121. Appl. Sci., 14.10.3390/app14177707 ↗
34. Missaoui, R., Hechkel, W., Saadaoui, W., Helali, A., and Leo, M. (2025). Advanced Deep Learning and Machine Learning Techniques for MRI Brain Tumor Analysis: A Review. Sensors, 25.10.3390/s25092746 ↗
35. Agarwal, D., Berbis, M.A., Martín-Noguerol, T., Luna, A., Garcia, S.C., and de la Tor-re-Díez, I. (2022). End-to-End Deep Learning Architectures Using 3D Neuroimaging Biomarkers for Early Alzheimer’s Diag-nosis. Mathematics, 10.10.3390/math10152575 ↗
36. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
37. Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., and Shpanskaya, K. (2017). Chexnet: Radiol-ogist-level pneumonia detection on chest x-rays with deep learning. arXiv.
38. Marengo, A., Pagano, A., and Santamato, V. (2024). An efficient cardiovascular disease prediction model through AI-driven IoT technology. Comput. Biol. Med., 183.10.1016/j.compbiomed.2024.109330 ↗
39. Santamato, V., Tricase, C., Faccilongo, N., Iacoviello, M., Pange, J., and Marengo, A. (2024). Machine Learning for Evaluating Hospital Mobility: An Italian Case Study. Appl. Sci., 14.10.20944/preprints202404.0006.v1 ↗
40. Healthcare performance analytics based on the novel PDA methodology for assessment of efficiency and perceived quality outcomes: A machine learning approach10.1016/j.eswa.2024.124020 ↗
41. Santamato, V., Tricase, C., Faccilongo, N., Iacoviello, M., and Marengo, A. (2024). Exploring the Impact of Artificial Intelligence on Healthcare Management: A Combined Systematic Review and Machine-Learning Approach. Appl. Sci., 14.10.3390/app142210144 ↗