Convolutional Neural Network and Language Model-Based Sequential CT Image Captioning for Intracerebral Hemorrhage
Source abstract
Abstract
Intracerebral hemorrhage is a severe problem where more than one-third of patients die within a month. In diagnosing intracranial hemorrhage, neuroimaging examinations are essential. As a result, the interpretation of neuroimaging becomes a crucial process in medical procedures. However, human-based image interpretation has inherent limitations, as it can only handle a restricted range of tasks. To address this, a study on medical image captioning has been conducted, but it primarily focused on single medical images. However, actual medical images often consist of continuous sequences, such as CT scans, making it challenging to directly apply existing studies. Therefore, this paper proposes a CT image captioning model that utilizes a 3D-CNN model and distilGPT-2. In this study, four combinations of 3D-CNN models and language models were compared and analyzed for their performance. Additionally, the impact of applying penalties to the loss function and adjusting penalty values during the training process was examined. The proposed CT image captioning model demonstrated a maximum BLEU score of 0.35 on the in-house dataset, and it was observed that the text generated by the model became more similar to human interpretations in medical image reports with the application of loss function penalties.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Intracerebral and Subarachnoid Hemorrhage Research · Acute Ischemic Stroke Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Convergence Software, Hallym University, Chuncheon-si 24252, Republic of Korea
Cerebrovascular Disease Research Center, Hallym University, Chuncheon-si 24252, Republic of Korea
Department of Neurology, Chuncheon Sacred Heart Hospital, Chuncheon-si 24252, Republic of Korea
Funding
National Research Foundation of Korea · Institute of Information & communications Technology Planning & Evaluation · Ministry of Health & Welfare, Republic of Korea
References (32)
Show source-provided references
1. Intracerebral haemorrhage: Current approaches to acute management10.1016/S0140-6736(18)31878-6 ↗
2. Global and regional burden of first-ever ischaemic and haemorrhagic stroke during 1990–2010: Findings from the Global Burden of Disease Study 201010.1016/S2214-109X(13)70089-5 ↗
3. Mohammed, B.A., Senan, E.M., Al-Mekhlafi, Z.G., Rassem, T.H., Makbol, N.M., Alanazi, A.A., Almurayziq, T.S., Ghaleb, F.A., and Sallam, A.A. (2022). Multi-Method Diagnosis of CT Images for Rapid Detection of Intracranial Hemorrhages Based on Deep and Hybrid Learning. Electronics, 11.10.3390/electronics11152460 ↗
4. Contemporary trends in the nationwide incidence of primary intracerebral hemorrhage10.1161/STROKEAHA.121.037332 ↗
5. Neuroimaging of Intracerebral Hemorrhage10.1093/neuros/nyaa029 ↗
6. London, T.R.C.O.R. (2023, July 21). Clinical Radiology UK Workforce Census 2020 Report. Available online: https://www.rcr.ac.uk/system/files/publication/field_publication_files/clinical-radiology-uk-workforce-census-2020-report.pdf.
7. Ionescu, B., Müller, H., Péteri, R., Rückert, J., Abacha, A.B., de Herrera, A.G.S., Friedrich, C.M., Bloch, L., Brüngel, R., and Idrissi-Yaghir, A. (2022). International Conference of the Cross-Language Evaluation Forum for European Languages, Springer International Publishing.
8. Hajihosseini, M., Lotfollahi, Y., Nobakhtian, M., Javid, M.M., Omidi, F., and Eetemadi, S. (2022, January 5–8). IUST_NLPLAB at ImageCLEFmedical Caption Tasks. Proceedings of the Conference and Labs of the Evaluation Forum, Bologna, Italy.
9. Lebrat, L., Nicolson, A., Santa Cruz, R., Belous, G., Koopman, B., and Dowling, J. (2022, January 5–8). CSIRO at ImageCLEFmedical Caption 2022. Proceedings of the CLEF 2022: Conference and Labs of the Evaluation Forum, Bologna, Italy.
10. Preparing a collection of radiology examinations for distribution and retrieval10.1093/jamia/ocv080 ↗
11. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports10.1038/s41597-019-0322-0 ↗
12. Medical image captioning via generative pretrained transformers10.1038/s41598-023-31223-5 ↗
13. Yang, S., Ji, J., Zhang, X., Liu, Y., and Wang, Z. (2021, January 9–12). Weakly Guided Hierarchical Encoder-Decoder Network for Brain CT Report Generation. Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Houston, TX, USA.10.1109/BIBM52615.2021.9669626 ↗
14. Intracranial Hemorrhage10.1016/j.emc.2012.06.003 ↗
15. Intracranial hemorrhage: Diagnosis and management10.1016/j.ncl.2011.09.002 ↗
16. A deep learning model for prognosis prediction after intracranial hemorrhage10.1111/jon.13078 ↗
17. Stroke classification from computed tomography scans using 3D convolutional neural network10.1016/j.bspc.2022.103720 ↗
18. Optimising a 3D convolutional neural network for head and neck computed tomography segmentation with limited training data10.1016/j.phro.2022.04.003 ↗
19. Rani, S., Kumar, S., Ghai, D., and Prasad, K. (2022, January 23–25). Automatic Detection of Brain Tumor from CT and MRI Images using Wireframe model and 3D Alex-Net. Proceedings of the 2022 International Conference on Decision Aid Sciences and Applications (DASA), Chiangrai, Thailand.10.1109/DASA54658.2022.9765114 ↗
20. He, K., Zhang, X., Ren, S., and Sun, J. (2016–1, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
21. Tan, M., and Le, Q. (2019, January 9–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
22. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
23. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
24. Olivastri, S., Singh, G., and Cuzzolin, F. (2019, January 16–17). End-to-end video captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Long Beach, CA, USA.10.1109/ICCVW.2019.00185 ↗
25. 3D convolutional neural networks for stalled brain capillary detection10.1016/j.compbiomed.2021.105089 ↗
26. Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., and Gonzalez, J. (2020, January 13–18). Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers. Proceedings of the 37th International Conference on Machine Learning, Virtual.
27. Mars, M. (2022). From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough. Appl. Sci., 12.10.3390/app12178805 ↗
28. Sharma, S., Asri, L.E., Schulz, H., and Zumer, J. (2017). Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. arXiv.
29. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA.10.3115/1073083.1073135 ↗
30. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop On intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
31. Lin, C.-Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
32. MedicalAI (2023, July 21). ClinicalBERT. Available online: https://huggingface.co/medicalai/ClinicalBERT.