From Label‐Free Multiphoton Imaging to Pathological Reports: A Vision‐Language Breast Cancer Margin Pathological Diagnosis System
Source abstract
Abstract
ABSTRACT Margin pathological assessment provides critical feedback for breast‐conserving surgery, whereas H&E‐stained histopathology focuses on residual tumor and may cause unnecessary resections. Label‐free multiphoton microscopy (MPM) reveals tumor‐associated collagen signatures (TACS) at the margin, offering a prognostically relevant diagnostic information. However, limited familiarity of novel MPM images by pathologists has prevented its integration into diagnostic workflows. Here, we introduce MarginPath, a Margin Pathological diagnosis system built on an MPM‐language model that requires only a single label‐free section. By integrating MPM‐derived TACS with virtual H&E diagnostic information, MarginPath provides a multimodal diagnostic report, including: (i) a MPM image and corresponding virtual H&E image, (ii) a TACS‐based pixel‐level margin‐status heatmap, and (iii) detailed natural‐language diagnostic descriptions of tumor margin microenvironment. Validated on 158 invasive breast cancer specimens, MarginPath outperforms existing pathology vision‐language models in margin diagnosis and can be extended into a question‐answering system, enhancing both clinical decision‐support and patient communication.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Fluorescence Microscopy Techniques · Cell Image Analysis Techniques · Photoacoustic and Ultrasonic Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering and Automation Fuzhou University Fuzhou China
National Biomedical Imaging Center State Key Laboratory of Membrane Biology Institute of Molecular Medicine Peking‐Tsinghua Center for Life Sciences College of Future Technology Peking University Beijing China
Fujian Provincial Key Laboratory of Photonics Technology Key Laboratory of OptoElectronic Science and Technology for Medicine of Ministry of Education Fujian Normal University Fuzhou China
College of Computer and Data Science Fuzhou University Fuzhou China
Department of Pathology Fujian Medical University Union Hospital Fuzhou China
Department of Pathology The First Affiliated Hospital of Fujian Medical University Fuzhou China
Department of Breast Surgery The Second Affiliated Hospital of Xiamen Medical College Xiamen China
State Key Laboratory of Photonics and Communications School of Electronics Peking University Beijing China
Department of Radiology Fujian Medical University Union Hospital Fuzhou China
Funding
National Natural Science Foundation of China · Natural Science Foundation of Fujian Province · Fujian Provincial Health Technology Project
References (58)
Show source-provided references
1. 10.1016/S0140‐6736(16)31891‐810.1016/S0140‐6736(16)31891‐8 ↗
2. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries
3. 10.1016/j.ejso.2016.08.01610.1016/j.ejso.2016.08.016 ↗
4. 10.1002/advs.20190151910.1002/advs.201901519 ↗
5. 10.1136/bmj‐2022‐07034610.1136/bmj‐2022‐070346 ↗
6. 10.1016/j.breast.2015.11.00310.1016/j.breast.2015.11.003 ↗
7. 10.1001/jamadermatol.2025.279010.1001/jamadermatol.2025.2790 ↗
8. Label‐Free Histopathology10.1038/nmeth.4008 ↗
9. 10.1038/s41377‐024‐01597‐w10.1038/s41377‐024‐01597‐w ↗
10. 10.1186/1741‐7015‐4‐3810.1186/1741‐7015‐4‐38 ↗
11. 10.1186/s12916‐021‐02146‐710.1186/s12916‐021‐02146‐7 ↗
12. 10.7150/thno.5592110.7150/thno.55921 ↗
13. 10.1038/s41467‐020‐20429‐010.1038/s41467‐020‐20429‐0 ↗
14. 10.1002/ijc.3485510.1002/ijc.34855 ↗
15. 10.1038/s41467‐022‐31771‐w10.1038/s41467‐022‐31771‐w ↗
16. Towards a Visual‐Language Foundation Model for Computational Pathology
17. 10.1038/s41591‐023‐02504‐310.1038/s41591‐023‐02504‐3 ↗
18. 10.1038/s41586‐024‐08378‐w10.1038/s41586‐024‐08378‐w ↗
19. 10.1038/s41586‐024‐08169‐310.1038/s41586‐024‐08169‐3 ↗
20. 10.1038/s41586‐024‐07618‐310.1038/s41586‐024‐07618‐3 ↗
21. Y.Chen G.Wang andY.Ji et al. “Slidechat: A Large Vision‐Language Assistant for Whole‐Slide Pathology Image Understanding ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(IEEE 2025) 5134–5143.10.1109/CVPR52734.2025.00484 ↗
22. 10.1038/s41467‐025‐60014‐x10.1038/s41467‐025‐60014‐x ↗
23. J.‐Y.Zhu T.Park P.Isola andA. A.Efros “Unpaired Image‐to‐Image Translation Using Cycle‐Consistent Adversarial Networks ” inProceedings of the IEEE International Conference on Computer Vision (ICCV)(IEEE 2017) 2223–2232.10.1109/ICCV.2017.244 ↗
24. 10.1016/j.ajpath.2010.11.07610.1016/j.ajpath.2010.11.076 ↗
25. 10.1007/s13402‐023‐00851‐410.1007/s13402‐023‐00851‐4 ↗
26. A.Dosovitskiy L.Beyer andA.Kolesnikov et al. “An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale ” inInternational Conference on Learning Representations(ICLR 2021).
27. Medmamba: Vision Mamba for Medical Image Classification
28. K.He X.Zhang S.Ren andJ.Sun “Deep Residual Learning for Image Recognition ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(IEEE 2016) 770–778.10.1109/CVPR.2016.90 ↗
29. J.Zhang X.Li andJ.Li “Rethinking Mobile Block for Efficient Attention‐Based Models ” in2023 IEEE/CVF International Conference on Computer Vision (ICCV)(IEEE 2023) 1389–1400 https://doi.org/10.1109/ICCV51070.2023.00134.10.1109/ICCV51070.2023.00134 ↗
30. Llava‐Med: Training a Large Language‐and‐Vision Assistant for Biomedicine in One Day
31. 10.1001/jamasurg.2020.302510.1001/jamasurg.2020.3025 ↗
32. Ductal Carcinoma In Situ and Margins <2 Mm: Contemporary Outcomes With Breast Conservation10.1097/SLA.0000000000002439 ↗
33. Breast Cancer, Screening and Diagnostic Tools: All You Need to Know10.1016/j.critrevonc.2020.103174 ↗
34. 10.1016/j.acra.2017.09.01810.1016/j.acra.2017.09.018 ↗
35. 10.1126/sciadv.160067510.1126/sciadv.1600675 ↗
36. 10.1158/0008‐5472.CAN‐20‐312410.1158/0008‐5472.CAN‐20‐3124 ↗
37. 10.1002/ijc.2603110.1002/ijc.26031 ↗
38. 10.1038/nrd425310.1038/nrd4253 ↗
39. 10.1038/nature1575610.1038/nature15756 ↗
40. 10.1172/jci.insight.15033010.1172/jci.insight.150330 ↗
41. Deep Learning‐Enabled Multiphoton Microscopy Predicts Colorectal Cancer Recurrence From Routine FFPE Specimens10.1038/s41746-025-02058-3 ↗
42. 10.1007/s10120‐024‐01551‐010.1007/s10120‐024‐01551‐0 ↗
43. 10.1038/s41467‐023‐41165‐110.1038/s41467‐023‐41165‐1 ↗
44. Gpt‐4o System Card
45. Generative Adversarial Nets
46. 10.1109/TIP.2003.81986110.1109/TIP.2003.819861 ↗
47. Attention Is All You Need
48. 10.1145/357173010.1145/3571730 ↗
49. Qwen2‐vl: Enhancing Vision‐Language Model's Perception of the World at any Resolution
50. I.LoshchilovandF.Hutter “Decoupled Weight Decay Regularization ” inInternational Conference on Learning Representations(ICLR 2019).
51. I.LoshchilovandF.Hutter “SGDR: Stochastic Gradient Descent With Warm Restarts ” inInternational Conference on Learning Representations(ICLR 2017).
52. Metrics for Multi‐Class Classification: An Overview
53. 10.1038/s41467‐018‐07619‐710.1038/s41467‐018‐07619‐7 ↗
54. Gemini 2.5: Pushing the Frontier With Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
55. Qwen2 Technical Report
56. K.Papineni S.Roukos T.Ward andW.‐J.Zhu “Bleu: A Method for Automatic Evaluation of Machine Translation ” inProceedings of the 40th Annual Meeting of the Association for Computational Linguistics(ACL 2002) 311–318.10.3115/1073083.1073135 ↗
57. C.‐Y.Lin “Rouge: A Package for Automatic Evaluation of Summaries ”Text Summarization Branches Out: Proceedings of the ACL‐04 Workshop(ACL 2004):74–81.
58. S.BanerjeeandA.Lavie “METEOR: An Automatic Metric for MT Evaluation With Improved Correlation With human Judgments ” inProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization(ACL 2005) 65–72.