Beyond Accuracy: A Multi-dimensional Cognitive Audit of Medical Large Vision–Language Models in Fundus Image Interpretation
Source abstract
Abstract
Reliance on standalone accuracy limits credible assessment of fundus-focused large vision–language models (LVLMs), as high scores often stem from linguistic shortcuts rather than real visual reasoning. This work develops the Cognitive Audit Framework (CAF), a four-module automated auditing pipeline that dissects model reasoning flaws: Visual–Linguistic Decoupling (textual dependency via modality ablation), Hierarchical Logical Consistency (lesion–diagnosis contradiction detection), Reasoning Fidelity Gap (chain-of-thought unfaithfulness scoring), and Contextual Robustness (positional bias under option permutation). Experiments on six 7B–31B LVLMs over FunBench reveal a notable gap between benchmark accuracy and reasoning quality: high accuracy coexists with measurable textual dependency, logical inconsistencies across diagnostic levels, limited chain-of-thought faithfulness, and non-trivial positional sensitivity. CAF serves as a reproducible complement to pure accuracy metrics for validating clinical competence of ophthalmic multimodal models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Retinal Imaging and Analysis · Multimodal Machine Learning Applications · Artificial Intelligence in Healthcare and Education
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Data Science, City University of Macau, Macau 999078, China
Institute of Artificial Intelligence, Putian University, Putian 351100, China
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
Funding
Science and Technology Development Fund · Fujian Province Young and Middle-aged Teachers’ Educational Science Research Projects
References (59)
Show source-provided references
1. A foundation model for generalizable disease detection from retinal images10.1038/s41586-023-06555-x ↗
2. Shi, D., Zhang, W., Yang, J., Huang, S., Chen, X., Yusufu, M., Jin, K., Lin, S., Liu, S., and Zhang, Q. (2024). Eyeclip: A visual-language foundation model for multi-modal ophthalmic image analysis. arXiv.10.1038/s41746-025-01772-2 ↗
3. Llava-med: Training a large language-and-vision assistant for biomedicine in one day
4. Towards generalist biomedical AI10.1056/AIoa2300138 ↗
5. Chen, Y., Xu, D., Huang, Y., Zhan, S., Wang, H., Chen, D., Wang, X., Qiu, M., and Li, H. (2025). MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output. Proceedings of the Computer Vision and Pattern Recognition Conference, IEEE.10.1109/CVPR52734.2025.02303 ↗
6. Liu, X., and Song, D. (2025). Constructing ophthalmic mllm for positioning-diagnosis collaboration through clinical cognitive chain reasoning. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51701.2025.02001 ↗
7. Enhancing diagnostic accuracy in rare and common fundus diseases with a knowledge-rich vision-language model10.1038/s41467-025-60577-9 ↗
8. Li, S., Lin, T., Lin, L., Zhang, W., Liu, J., Yang, X., Li, J., He, Y., Song, X., and Xiao, J. (2025). Eyecaregpt: Boosting comprehensive ophthalmology understanding with tailored dataset, benchmark and model. Proceedings of the 33rd ACM International Conference on Multimedia, Association for Computing Machinery.10.1145/3746027.3755229 ↗
9. Zhou, T., Xu, Y., Zhu, Y., Xiao, C., Bian, H., Wei, L., and Zhang, X. (2026). DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?. Advances in Neural Information Processing Systems, Curran Associates, Inc.
10. Shortcut learning in deep neural networks10.1038/s42256-020-00257-z ↗
11. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales10.1016/S0161-6420(03)00475-5 ↗
12. Early Treatment Diabetic Retinopathy Study Research Group (1991). Grading diabetic retinopathy from stereoscopic color fundus photographs—An extension of the modified Airlie House classification: ETDRS report number 10. Ophthalmology, 98, 786–806.10.1016/S0161-6420(13)38012-9 ↗
13. U.S. Food and Drug Administration (FDA) (2021). Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan.
14. TRIPOD+ AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods10.1136/bmj-2023-078378 ↗
15. Judging llm-as-a-judge with mt-bench and chatbot arena
16. Jain, S., Agrawal, A., Saporta, A., Truong, S.Q., Duong, D.N., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., and Ng, A.Y. (2021). Radgraph: Extracting clinical entities and relations from radiology reports. arXiv.
17. Green: Generative radiology report evaluation and error notation
18. Huang, A., Banerjee, O., Wu, K., Reis, E.P., and Rajpurkar, P. (2024). Fineradscore: A radiology report line-by-line evaluation technique generating corrections with severity scores. arXiv.
19. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting
20. Ghosh, B., Hasan, S., Arafat, N.A., and Khan, A. (2024). Logical consistency of large language models in fact-checking. arXiv.
21. Wei, Q., Qian, K., and Li, X. (2025). FunBench: Benchmarking fundus reading skills of MLLMs. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-032-04978-0_27 ↗
22. High-performance medicine: The convergence of human and artificial intelligence10.1038/s41591-018-0300-7 ↗
23. Identifying medical diagnoses and treatable diseases by image-based deep learning10.1016/j.cell.2018.02.010 ↗
24. Large language models in medicine10.1038/s41591-023-02448-8 ↗
25. Liu, C., Pan, J., Shen, W., Bai, W., Rueckert, D., and Arcucci, R. (2025). How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study. arXiv.
26. Visionunite: A vision-language foundation model for ophthalmology enhanced with clinical knowledge10.1109/TPAMI.2025.3598734 ↗
27. Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., and Liu, Z. (2024). Mmbench: Is your multi-modal model an all-around player?. Proceedings of the European Conference on Computer Vision, Springer.10.1007/978-3-031-72658-3_13 ↗
28. A dataset of clinically generated visual questions and answers about radiology images10.1038/sdata.2018.251 ↗
29. Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., and Wu, X.M. (2021). Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), IEEE.10.1109/ISBI48211.2021.9434010 ↗
30. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction10.1148/radiol.2017171920 ↗
31. Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., and Lungren, M. (2020). Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics (ACL).10.18653/v1/2020.emnlp-main.117 ↗
32. Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., and Xie, W. (2023). Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv.
33. Liu, B., Zou, K., Zhan, L.M., Lu, Z., Dong, X., Chen, Y., Xie, C., Cao, J., Wu, X.M., and Fu, H. (2025). Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51701.2025.01979 ↗
34. Ko, M., Lee, J., Kim, H., Kim, G., and Kang, J. (2020). Look at the first sentence: Position bias in question answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics (ACL).10.18653/v1/2020.emnlp-main.84 ↗
35. Large language models sensitivity to the order of options in multiple-choice questions
36. Large language models are not robust multiple choice selectors
37. Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects
38. Kang, S., Kim, J., Kim, J., and Hwang, S.J. (2025). Your large vision-language model only needs a few attention heads for visual grounding. Proceedings of the Computer Vision and Pattern Recognition Conference, IEEE.10.1109/CVPR52734.2025.00872 ↗
39. Xu, Y., Zhu, L., and Yang, Y. (2025). Mc-bench: A benchmark for multi-context visual grounding in the era of mllms. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51701.2025.01642 ↗
40. Chain-of-thought prompting elicits reasoning in large language models
41. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv.
42. Lin, G.Y., Sun, C.E., and Weng, T.W. (2025, January 6). FoCus: Improving Faithfulness in Chain-of-Thoughts by Training on Structured Reasoning Data. Proceedings of the the 5th Workshop on Mathematical Reasoning and AI at NeurIPS, San Diego, CA, USA.
43. Logically consistent language models via neuro-symbolic integration
44. Liu, Y., Guo, Z., Liang, T., Shareghi, E., Vulić, I., and Collier, N. (2024). Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models. arXiv.
45. Bansal, A., and Gangwani, I. (2026, January 26–27). Logical Consistency Under Pressure: Probing and Repairing Cross-Query Contradictions in LLMs. Proceedings of the ICLR 2026 Workshop on Logical Reasoning of Large Language Models, Rio de Janeiro, Brazil.
46. Sun, Z., Wang, Q., Wang, H., Zhang, X., and Xu, J. (2025). Detection and mitigation of hallucination in large reasoning models: A mechanistic perspective. arXiv.
47. Causality: Models, reasoning, and inference10.1007/s11615-001-0048-3 ↗
48. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening10.1016/j.ins.2019.06.011 ↗
49. DeepDRiD: Diabetic retinopathy—Grading and image quality estimation challenge10.1016/j.patter.2022.100512 ↗
50. Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, F. (2018). Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data, 3.10.3390/data3030025 ↗
51. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks10.1038/s41467-021-25138-w ↗
52. Learning two-stream CNN for multi-modal age-related macular degeneration categorization10.1109/JBHI.2022.3171523 ↗
53. Multi-scale convolutional neural network for automated AMD classification using retinal OCT images10.1016/j.compbiomed.2022.105368 ↗
54. Octdl: Optical coherence tomography dataset for image-based deep learning methods10.1038/s41597-024-03182-7 ↗
55. OCTID: Optical coherence tomography image database10.1016/j.compeleceng.2019.106532 ↗
56. Li, N., Li, T., Hu, C., Wang, K., and Kang, H. (2020). A benchmark of ocular disease intelligent recognition: One shot for multi-disease detection. Proceedings of the International Symposium on Benchmarking, Measuring and Optimization, Springer.10.1007/978-3-030-71058-3_11 ↗
57. Wei, Q., Li, X., Yu, W., Zhang, X., Zhang, Y., Hu, B., Mo, B., Gong, D., Chen, N., and Ding, D. (2021). Learn to segment retinal lesions and beyond. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), IEEE.10.1109/ICPR48806.2021.9412088 ↗
58. RETOUCH: The retinal OCT fluid detection and segmentation benchmark and challenge10.1109/TMI.2019.2901398 ↗
59. Pachade, S., Porwal, P., Thulkar, D., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., Giancardo, L., Quellec, G., and Mériaudeau, F. (2021). Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research. Data, 6.10.3390/data6020014 ↗