ACapMed: Automatic Captioning for Medical Imaging
Source abstract
Abstract
Medical image captioning is a very challenging task that has been rarely addressed in the literature on natural image captioning. Some existing image captioning techniques exploit objects present in the image next to the visual features while generating descriptions. However, this is not possible for medical image captioning when one requires following clinician-like explanations in image content descriptions. Inspired by the preceding, this paper proposes using medical concepts associated with images, in accordance with their visual features, to generate new captions. Our end-to-end trainable network is composed of a semantic feature encoder based on a multi-label classifier to identify medical concepts related to images, a visual feature encoder, and an LSTM model for text generation. Beam search is employed to ensure the best selection of the next word for a given sequence of words based on the merged features of the medical image. We evaluated our proposal on the ImageCLEF medical captioning dataset, and the results demonstrate the effectiveness and efficiency of the developed approach.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Machine Vision and Signal Analysis, University of Oulu, FI-90014 Oulu, Finland
Faculty of Medicine, University of Oulu, FI-90014 Oulu, Finland
IDP Laboratory—UMR CNRS 7013, University of Orleans, CEDEX 2, 45067 Orleans, France
Funding
Academy of Finland Profi5 DigiHealth project
References (28)
Show source-provided references
1. Generating diagnostic report for medical image by high-middle-level visual information incorporation on double deep learning models10.1016/j.cmpb.2020.105700 ↗
2. Unifying neural learning and symbolic reasoning for spinal medical report generation10.1016/j.media.2020.101872 ↗
3. Beddiar, D.R., Oussalah, M., and Seppänen, T. (2022). Automatic captioning for medical imaging (MIC): A rapid review of literature. Artif. Intell. Rev.10.1007/s10462-022-10270-w ↗
4. Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
5. Automatic ultrasound image report generation with adaptive multimodal attention mechanism10.1016/j.neucom.2020.09.084 ↗
6. Li, C., Liang, X., Hu, Z., Xing, E., Li, C.Y., Liang, X., Hu, Z., and Xing, E.P. (2018). Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation. Advances in Neural Information Processing Systems 31 (NIPS 2018), Proceedings of the Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montreal, QC, Canada, 3–8 December 2018, Curran Associates, Inc.
7. Kumar, D., Srivastava, V., Popescu, D.E., and Hemanth, J.D. (2022). Dual-Modal Transformer with Enhanced Inter-and Intra-Modality Interactions for Image Captioning. Appl. Sci., 12.10.3390/app12136733 ↗
8. Chang, Y.H., Chen, Y.J., Huang, R.H., and Yu, Y.T. (2021). Enhanced Image Captioning with Color Recognition Using Deep Learning Methods. Appl. Sci., 12.10.3390/app12010209 ↗
9. Understanding and Generating Ultrasound Image Description10.1007/s11390-018-1874-8 ↗
10. Automatic medical image interpretation: State of the art and future directions10.1016/j.patcog.2021.107856 ↗
11. Onita, D., Birlutiu, A., and Dinu, L. (2020). Towards mapping images to text using deep-learning architectures. Mathematics, 8.10.3390/math8091606 ↗
12. Medical Image Labelling and Semantic Understanding for Clinical Applications
13. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
14. Yin, C., Qian, B., Wei, J., Li, X., Zhang, X., Li, Y., Zheng, Q., Yin, C., Qian, B., and Wei, J. (2019, January 8–11). Automatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network. Proceedings of the 2019 19th IEEE International Conference on Data Mining (ICDM 2019), Beijing, China.10.1109/ICDM.2019.00083 ↗
15. Beddiar, D.R., Oussalah, M., and Seppänen, T. (2021, January 21–24). Attention-based CNN-GRU model for automatic medical images captioning: ImageCLEF 2021. Proceedings of the Working Notes of CLEF 2021—Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania.
16. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 7–9). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, PMLR, Lille, France.
17. A Computational Framework Towards Medical Image Explanation
18. Al Duhayyim, M., Alazwari, S., Mengash, H.A., Marzouk, R., Alzahrani, J.S., Mahgoub, H., Althukair, F., and Salama, A.S. (2022). Metaheuristics Optimization with Deep Learning Enabled Automated Image Captioning System. Appl. Sci., 12.10.3390/app12157724 ↗
19. Xie, X., Xiong, Y., Yu, P., Li, K., Zhang, S., and Zhu, Y. (2019, January 22–25). Attention-Based Abnormal-Aware Fusion Network for Radiology Report Generation. Proceedings of the 24th International Conference on Database Systems for Advanced Applications, DASFAA 2019, Chiang Mai, Thailand.10.1007/978-3-030-18590-9_64 ↗
20. Li, C.Y., Liang, X., Hu, Z., and Xing, E.P. (2019, January 8–12). Knowledge-driven encode, retrieve, paraphrase for medical image report generation. Proceedings of the AAAI Conference on Artificial Intelligence, Atlanta, GA, USA.10.1609/aaai.v33i01.33016666 ↗
21. Zhao, S., Li, L., Peng, H., Yang, Z., and Zhang, J. (2020). Image Caption Generation via Unified Retrieval and Generation-Based Method. Appl. Sci., 10.10.3390/app10186235 ↗
22. Pelka, O., Ben Abacha, A., García Seco de Herrera, A., Jacutprakart, J., Friedrich, C.M., and Müller, H. (2021, January 21–24). Overview of the ImageCLEFmed 2021 Concept & Caption Prediction Task. Proceedings of the CLEF2021 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania.
23. Ionescu, B., Müller, H., Peteri, R., Abacha, A.B., Sarrouti, M., Demner-Fushman, D., Hasan, S.A., Kozlovski, S., Liauchuk, V., and Dicente, Y. (2021). Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications. LNCS Lecture Notes in Computer Science, Proceedings of the 12th International Conference of the CLEF Association (CLEF 2021), Bucharest, Romania, 21–24 September 2021, Springer.10.1007/978-3-030-85251-1_23 ↗
24. Castro, V., Pino, P., Parra, D., and Lobel, H. (2021, January 21–24). PUC Chile team at Caption Prediction: ResNet visual encoding and caption classification with Parametric ReLU. Proceedings of the CLEF (Working Notes), Bucharest, Romania.
25. Wang, X., Guo, Z., Xu, C., Sun, L., and Li, J. (2021, January 21–24). ImageSem Group at ImageCLEFmed Caption 2021 Task: Exploring the Clinical Significance of the Textual Descriptions Derived from Medical Images. Proceedings of the CLEF (Working Notes), Bucharest, Romania.
26. Tsuneda, R., Asakawa, T., and Aono, M. (2021, January 21–24). Kdelab at ImageCLEF 2021: Medical Caption Prediction with Effective Data Pre-processing and Deep Learning. Proceedings of the CLEF (Working Notes), Bucharest, Romania.
27. Schuit, G., Castro, V., Pino, P., Parra, D., and Lobel, H. (2021, January 21–24). PUC Chile team at Concept Detection: K Nearest Neighbors with Perceptual Similarity. Proceedings of the CLEF (Working Notes), Bucharest, Romania.
28. Jacutprakart, J., Andrade, F.P., Cuan, R., Compean, A.A., Papanastasiou, G., and de Herrera, A.G.S. (2021, January 21–24). NLIP-Essex-ITESM at ImageCLEFcaption 2021 task: Deep Learning-based Information Retrieval and Multi-label Classification towards improving Medical Image Understanding. Proceedings of the CLEF (Working Notes), Bucharest, Romania.