Graph-Driven Medical Report Generation with Adaptive Knowledge Distillation
Source abstract
Abstract
Automated medical report generation (MRG) faces a critical hurdle in seamlessly integrating detailed visual evidence with accurate clinical diagnoses. Current approaches often rely on static knowledge transfer, overlooking the complex interdependencies among pathological findings and their nuanced alignment with visual evidence, often yielding reports that are linguistically sound but clinically misaligned. To address these limitations, we propose a novel graph-driven medical report generation framework with adaptive knowledge distillation. Our architecture leverages a dual-phase optimization process. First, visual–semantic enhancement proceeds through the explicit correlation of image features with a structured knowledge network and their concurrent enrichment via cross-modal semantic fusion, ensuring that generated descriptions are grounded in anatomical and pathological context. Second, a knowledge distillation mechanism iteratively refines both global narrative flow and local descriptive precision, enhancing the consistency between images and text. Comprehensive experiments on the MIMIC-CXR and IU X-Ray datasets demonstrate the effectiveness of our approach, which achieves state-of-the-art performance in clinical efficacy metrics across both datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Multimodal Machine Learning Applications · Advanced Graph Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University, Hangzhou 310018, China
Zhejiang Provincial Innovation Center of Advanced Textile Technology, Shaoxing 312000, China
Funding
Key Research and Development Programs of Zhejiang Province · Science Foundation of Zhejiang Sci-Tech University
References (55)
Show source-provided references
1. Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., and Chang, X. (2022, January 18–24). Cross-modal clinical graph transformer for ophthalmic report generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.02000 ↗
2. Li, Y., Yang, B., Cheng, X., Zhu, Z., Li, H., and Zou, Y. (2023, January 1–6). Unify, align and refine: Multi-level semantic alignment for radiology report generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00268 ↗
3. Wang, J., Bhalerao, A., and He, Y. (2022, January 23–27). Cross-modal prototype driven network for radiology report generation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19833-5_33 ↗
4. Wang, Z., Liu, L., Wang, L., and Zhou, L. (2023, January 1–6). Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Paris, France.10.1109/CVPR52729.2023.01112 ↗
5. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports10.1038/s42256-021-00425-9 ↗
6. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2017, January 21–26). ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.369 ↗
7. Preparing a collection of radiology examinations for distribution and retrieval10.1093/jamia/ocv080 ↗
8. Lebret, R., Grangier, D., and Auli, M. (2016). Neural Text Generation from Structured Data with Application to the Biography Domain. arXiv.10.18653/v1/D16-1128 ↗
9. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
10. Lin, C.Y. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text summarization Branches Out, Barcelona, Spain.
11. Denkowski, M., and Lavie, A. (2011, January 30). Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, UK.
12. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–23). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
13. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020, January 13–19). Meshed-memory transformer for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01059 ↗
14. Hirota, Y., Nakashima, Y., and Garcia, N. (2022, January 18–24). Quantifying societal bias amplification in image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01309 ↗
15. Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., and Wang, L. (2022, January 18–24). Scaling up vision-language pre-training for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01745 ↗
16. Liu, F., Ge, S., Zou, Y., and Wu, X. (2022). Competence-based multimodal curriculum learning for medical report generation. arXiv.10.18653/v1/2021.acl-long.234 ↗
17. Chen, Z., Song, Y., Chang, T.H., and Wan, X. (2020). Generating radiology reports via memory-driven transformer. arXiv.10.18653/v1/2020.emnlp-main.112 ↗
18. Chen, Z., Shen, Y., Song, Y., and Wan, X. (2022). Cross-modal memory networks for radiology report generation. arXiv.10.18653/v1/2021.acl-long.459 ↗
19. Auxiliary signal-guided knowledge encoder-decoder for medical report generation10.1007/s11280-022-01013-6 ↗
20. Knowledge matters: Chest radiology report generation with general and specific knowledge10.1016/j.media.2022.102510 ↗
21. Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., and Xu, D. (2020, January 7–12). When radiology report generation meets knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6989 ↗
22. Liu, F., Wu, X., Ge, S., Fan, W., and Zou, Y. (2021, January 20–25). Exploring and distilling posterior and prior knowledge for radiology report generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01354 ↗
23. Jing, B., Xie, P., and Xing, E. (2017). On the automatic generation of medical imaging reports. arXiv.10.18653/v1/P18-1240 ↗
24. Automated radiographic report generation purely on transformer: A multicriteria supervised approach10.1109/TMI.2022.3171661 ↗
25. Yan, B., and Pei, M. (March, January 22). Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
26. MIMIC-CXR Database (version 2.1.0)
27. Liu, W., Xue, Y., Lin, C., and Boumaraf, S. (2025, October 09). Dataset: IU X-Ray Dataset. Available online: https://service.tib.eu/ldmservice/dataset/iu-x-ray-dataset.
28. Gu, J., Wang, G., Cai, J., and Chen, T. (2017, January 22–29). An empirical study of language cnn for image captioning. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.138 ↗
29. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
30. Huang, L., Wang, W., Chen, J., and Wei, X.Y. (November, January 27). Attention on attention for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, South Korea.
31. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017, January 21–26). Self-critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.131 ↗
32. Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., and Ji, R. (2021, January 20–25). RSTNet: Captioning with adaptive attention on visual and non-visual words. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01521 ↗
33. Yang, X., Tang, K., Zhang, H., and Cai, J. (2019, January 15–20). Auto-encoding scene graphs for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01094 ↗
34. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_42 ↗
35. Li, J., Li, D., Xiong, C., and Hoi, S. (2022, January 17–23). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the International Conference on Machine Learning, Baltimore, Maryland, USA.
36. Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (2022). Coca: Contrastive captioners are image-text foundation models. arXiv.
37. Liu, G., Hsu, T.M.H., McDermott, M., Boag, W., Weng, W.H., Szolovits, P., and Ghassemi, M. (2019, January 9–10). Clinically accurate chest x-ray report generation. Proceedings of the Machine Learning for Healthcare Conference, Ann Arbor, Michigan, USA.
38. Xue, Y., Xu, T., Rodney Long, L., Xue, Z., Antani, S., Thoma, G.R., and Huang, X. (2018, January 16–20). Multimodal recurrent model with attention for automated radiology report generation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain.10.1007/978-3-030-00928-1_52 ↗
39. Xue, Y., and Huang, X. (2019, January 2–7). Improved disease classification in chest x-rays with transferred features from report generation. Proceedings of the International Conference on Information Processing in Medical Imaging, Hong Kong, China.10.1007/978-3-030-20351-1_10 ↗
40. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
41. Hou, B., Kaissis, G., Summers, R.M., and Kainz, B. (October, January 27). Ratchet: Medical transformer for chest x-ray diagnosis and reporting. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Strasbourg, France.
42. Wang, Z., Tang, M., Wang, L., Li, X., and Zhou, L. (2022, January 18–22). A medical semantic-assisted transformer for radiographic report generation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore.10.1007/978-3-031-16437-8_63 ↗
43. Tanida, T., Müller, P., Kaissis, G., and Rueckert, D. (2023, January 17–24). Interactive and explainable region-guided radiology report generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00718 ↗
44. Endo, M., Krishnan, R., Krishna, V., Ng, A.Y., and Rajpurkar, P. (2021, January 4). Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model. Proceedings of the Machine Learning for Health, Online.
45. Jin, H., Che, H., Lin, Y., and Chen, H. (2024, January 26–27). Promptmrg: Diagnosis-driven prompts for medical report generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver Convention Center, Vancouver, BC, Canada.10.1609/aaai.v38i3.28038 ↗
46. Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., and Lungren, M.P. (2020). CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv.10.18653/v1/2020.emnlp-main.117 ↗
47. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports10.1038/s41597-019-0322-0 ↗
48. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals10.1161/01.CIR.101.23.e215 ↗
49. Improving chest X-ray report generation by leveraging warm starting10.1016/j.artmed.2023.102633 ↗
50. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
51. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota. Volume 1 (Long and Short Papers).
52. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
53. Radiology report generation with a learned knowledge base and multi-modal alignment10.1016/j.media.2023.102798 ↗
54. Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., and Chang, X. (2023, January 17–24). Dynamic graph enhanced contrastive learning for chest x-ray report generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00325 ↗
55. Huang, Z., Zhang, X., and Zhang, S. (2023, January 17–24). Kiut: Knowledge-injected u-transformer for radiology report generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01897 ↗