His‐MMDM: Multi‐Domain and Multi‐Omics Translation of Histopathological Images with Diffusion Models
Source abstract
Abstract
ABSTRACT Generative AI (GenAI) has advanced computational pathology through various image translation models. These models synthesize histopathological images from existing ones, facilitating tasks such as color normalization and virtual staining. Current models, while effective, are mostly dedicated to specific source‐target domain pairs and lack scalability for multi‐domain translations. Here, we introduce His‐MMDM, a diffusion model‐based framework enabling multi‐domain and multi‐omics histopathological image translation. His‐MMDM is not only effective in performing existing tasks such as transforming cryosectioned images to FFPE ones and virtual immunohistochemical (IHC) staining but can also facilitate knowledge transfer between different tumor types and between primary and metastatic tumors. Additionally, it performs genomics‐ and/or transcriptomics‐guided editing of histopathological images, illustrating the impact of driver mutations and oncogenic pathway alterations on tissue histopathology and educating pathologists to recognize them. These versatile capabilities position His‐MMDM as a versatile tool in the GenAI toolkit for future pathologists.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: AI in cancer detection · Cell Image Analysis Techniques · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division King Abdullah University of Science and Technology (KAUST) Thuwal Saudi Arabia
Center of Excellence on Smart Health King Abdullah University of Science and Technology (KAUST) Thuwal Saudi Arabia
Department of Life Science Dezhou University Dezhou China
Department of Neurosurgery Harbin Medical University Cancer Hospital Harbin China
Department of Pathology The First Affiliated Hospital of Harbin Medical University Harbin China
Department of Neurosurgery The First Affiliated Hospital of Harbin Medical University Harbin China
Department of Pathology Harbin Medical University Cancer Hospital Harbin China
Funding
King Abdullah University of Science and Technology
References (95)
Show source-provided references
1. R.Rombach A.Blattmann D.Lorenz P.Esser andB.Ommer “High‐resolution Image Synthesis with Latent Diffusion Models ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE2022):10674–10685 https://doi.org/10.1109/CVPR52688.2022.01042.10.1109/CVPR52688.2022.01042 ↗
2. Midjourney Inc. Midjourney accessed July 12 2024 www.midjourney.com2022.
3. Improving Image Generation with Better Captions
4. The Dawn of Lmms: Preliminary Explorations with gpt‐4v (ision)
5. L.Zhang A.Rao andM.Agrawala “Adding Conditional Control to Text‐to‐image Diffusion Models ” inProceedings of the IEEE/CVF International Conference on Computer Vision (IEEE 2023):3813–3824 https://doi.org/10.1109/ICCV51070.2023.00355.10.1109/ICCV51070.2023.00355 ↗
6. I.Goodfellow J.Pouget‐Abadie M.Mirza et al. “Generative Adversarial Nets ” inNeurIPS Proceedings (NIPS 2014).
7. Denoising Diffusion Probabilistic Models
8. Latent Transcriptional Programs Reveal Histology‐Encoded Tumor Features Spanning Tissue Origins
9. P. A.Moghadam S. V.Dalen K. C.Martin et al. “A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images ” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (IEEE 2023) https://doi.org/10.1109/WACV56688.2023.00204.10.1109/WACV56688.2023.00204 ↗
10. NASDM: Nuclei‐Aware Semantic Histopathology Image Generation Using Diffusion Models
11. W. S.Wong M.Amer T.Maul I. Y.Liao andA.Ahmed “Conditional Generative Adversarial Networks for Data Augmentation in Breast Cancer Classification ” inRecent Advances on Soft Computing and Data Mining: Proceedings of the Fourth International Conference on Soft Computing and Data Mining (SCDM 2020) (Springer 2020).10.1007/978-3-030-36056-6_37 ↗
12. 10.1016/j.bspc.2019.10178210.1016/j.bspc.2019.101782 ↗
13. Y.Xue Q.Zhou J.Ye et al. “Synthetic Augmentation and Feature‐based Filtering for Improved Cervical Histopathology Image Classification ” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference (Springer 2019).10.1007/978-3-030-32239-7_43 ↗
14. 10.1038/s41591‐023‐02504‐310.1038/s41591‐023‐02504‐3 ↗
15. 10.1038/s41591‐024‐02857‐310.1038/s41591‐024‐02857‐3 ↗
16. 10.1038/s41591‐024‐02856‐410.1038/s41591‐024‐02856‐4 ↗
17. 10.1038/s41586-024-07618-310.1038/s41586-024-07618-3 ↗
18. 10.1038/s41467‐024‐50043‐310.1038/s41467‐024‐50043‐3 ↗
19. F. G.Zanjani S.Zinger B. E.Bejnordi andJ.AWM van der Laak “Histopathology Stain‐color Normalization Using Deep Generative Models ” in1st Conference on Medical Imaging with Deep Learning (MIDL 2018) (MIDL 2018).
20. 10.1038/s41467‐021‐25221‐210.1038/s41467‐021‐25221‐2 ↗
21. 10.1038/s41551‐019‐0362‐y10.1038/s41551‐019‐0362‐y ↗
22. 10.1038/s42256‐022‐00471‐x10.1038/s42256‐022‐00471‐x ↗
23. 10.1038/s42256‐024‐00889‐510.1038/s42256‐024‐00889‐5 ↗
24. 10.1038/s41551‐022‐00952‐910.1038/s41551‐022‐00952‐9 ↗
25. 10.1126/sciadv.adn342610.1126/sciadv.adn3426 ↗
26. J.‐Y.Zhu T.Park P.Isola andA. A.Efros “Unpaired Image‐to‐Image Translation Using Cycle‐Consistent Adversarial Networks ” inProceedings of the IEEE International Conference on Computer Vision (IEEE 2017):2242–2251.10.1109/ICCV.2017.244 ↗
27. Diffusion Models Beat Gans on Image Synthesis
28. X.Su J.Song C.Meng andS.Ermon “Dual Diffusion Implicit Bridges for Image‐to‐Image Translation ”arXiv preprint(2022): 2203.08382.
29. 10.1038/s41591‐023‐02530‐110.1038/s41591‐023‐02530‐1 ↗
30. 10.1038/s41586‐024‐07441‐w10.1038/s41586‐024‐07441‐w ↗
31. 10.1038/s41586‐024‐07894‐z10.1038/s41586‐024‐07894‐z ↗
32. Ronneberger O. · 2015
33. J.Song C.Meng andS.Ermon “Denoising Diffusion Implicit Models ”arXiv preprint(2021): 2010.02502.
34. T.Park A. A.Efros R.Zhang andJ. Y.Zhu “Contrastive Learning for Unpaired Image‐to‐Image Translation ” inEuropean Conference on Computer Vision (Cham: Springer International Publishing 2020) 319–345.10.1007/978-3-030-58545-7_19 ↗
35. T.Karras S.Laine M.Aittala J.Hellsten J.Lehtinen andT.Aila “Analyzing and Improving the Image Quality of Stylegan ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE 2020):8107–8116 https://doi.org/10.1109/CVPR42600.2020.00813.10.1109/CVPR42600.2020.00813 ↗
36. D2c: Diffusion‐Decoding Models for Few‐Shot Conditional Generation
37. A.Radford J. W.Kim C.Hallacy andA.Ramesh “Learning Transferable Visual Models from Natural Language Supervision ” inInternational Conference on Machine Learning (PMLR 2021):8748–8763.
38. C.Szegedy V.Vanhoucke S.Ioffe J.Shlens andZ.Wojna “Rethinking the Inception Architecture for Computer Vision ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE 2016 2818–2826 https://doi.org/10.1109/CVPR.2016.308.10.1109/CVPR.2016.308 ↗
39. 10.1016/j.crmeth.2022.10019110.1016/j.crmeth.2022.100191 ↗
40. L. A.Gatys A. S.Ecker andM.Bethge “Image Style Transfer Using Convolutional Neural Networks ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE 2016):2414–2423 https://doi.org/10.1109/CVPR.2016.265.10.1109/CVPR.2016.265 ↗
41. 10.1038/s41467‐020‐20030‐510.1038/s41467‐020‐20030‐5 ↗
42. 10.3389/fonc.2022.84275910.3389/fonc.2022.842759 ↗
43. 10.1038/s41586‐021‐03512‐410.1038/s41586‐021‐03512‐4 ↗
44. 10.1038/s41591‐024‐02915‐w10.1038/s41591‐024‐02915‐w ↗
45. 10.1111/j.1365‐2559.2006.02472.x10.1111/j.1365‐2559.2006.02472.x ↗
46. 10.1038/s41467‐021‐21674‐710.1038/s41467‐021‐21674‐7 ↗
47. 10.1038/s41467‐022‐30746‐110.1038/s41467‐022‐30746‐1 ↗
48. 10.1038/s43856‐023‐00282‐010.1038/s43856‐023‐00282‐0 ↗
49. F. T.Liu K. M.Ting andZ. H.Zhou “Isolation Forest ” in2008 Eighth IEEE International Conference on Data Mining (IEEE 2008):413–422 https://doi.org/10.1109/ICDM.2008.17.10.1109/ICDM.2008.17 ↗
50. Comparison of Immunohistochemistry and DNA Sequencing for the Detection of IDH1 Mutations in Gliomas
51. 10.1016/S2589‐7500(21)00180‐110.1016/S2589‐7500(21)00180‐1 ↗
52. 10.1038/s41698‐020‐0120‐310.1038/s41698‐020‐0120‐3 ↗
53. 10.1038/s42256‐020‐0190‐510.1038/s42256‐020‐0190‐5 ↗
54. 10.1038/s41591‐018‐0177‐510.1038/s41591‐018‐0177‐5 ↗
55. 10.1038/s41698‐021‐00225‐910.1038/s41698‐021‐00225‐9 ↗
56. 10.1016/j.isci.2022.10587210.1016/j.isci.2022.105872 ↗
57. 10.1038/s43018‐020‐0085‐810.1038/s43018‐020‐0085‐8 ↗
58. 10.1038/s41467‐020‐17678‐410.1038/s41467‐020‐17678‐4 ↗
59. 10.1016/j.cell.2018.03.03510.1016/j.cell.2018.03.035 ↗
60. 10.1073/pnas.050658010210.1073/pnas.0506580102 ↗
61. 10.1016/j.cels.2015.12.00410.1016/j.cels.2015.12.004 ↗
62. 10.1016/j.media.2019.10156310.1016/j.media.2019.101563 ↗
63. 10.1038/s41467‐024‐46370‐010.1038/s41467‐024‐46370‐0 ↗
64. 10.1038/s41586-020-1969-610.1038/s41586-020-1969-6 ↗
65. 10.1016/j.immuni.2018.03.02310.1016/j.immuni.2018.03.023 ↗
66. 10.1038/nm.396710.1038/nm.3967 ↗
67. 10.1038/s41698‐023‐00399‐410.1038/s41698‐023‐00399‐4 ↗
68. 10.1016/j.crmeth.2023.10053410.1016/j.crmeth.2023.100534 ↗
69. 10.1038/s41551‐024‐01193‐810.1038/s41551‐024‐01193‐8 ↗
70. AutoGPT accessed July 12 2024 https://news.agpt.co/2023.
71. Y.Cao “A Comprehensive Survey of Ai‐Generated Content (aigc): A History of Generative ai from gan to Chatgpt ”arXiv preprint(2023): 2303.04226.
72. B.Roziere J.Gehring F.Gloeckle et al. “Code Llama: Open Foundation Models for Code ”arXiv preprint(2023): 2308.12950.
73. M. Y.Lu B.Chen D. F. K.Williamson et al. “A Foundational Multimodal Vision Language AI Assistant for Human Pathology ”arXiv preprint(2023): 2312.07814.
74. An AI Agent for Fully Automated Multi‐Omic Analyses
75. 10.1038/s41598‐017‐17204‐510.1038/s41598‐017‐17204‐5 ↗
76. Image Processing with ImageJ
77. Toolformer: Language Models Can Teach Themselves to Use Tools
78. A.Ulhaq N.Akhtar andG.Pogrebna “Efficient Diffusion Models for Vision: a Survey ”arXiv preprint(2022): 2210.09292.
79. Dpm‐Solver: A Fast Ode Solver for Diffusion Probabilistic Model Sampling in around 10 Steps
80. C.Lu Y.Zhou F.Bao J.Chen C.Li andJ.Zhu “Dpm‐Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models ”arXiv preprint(2022): 2211.01095.
81. The UCSC Xena Platform for Public and Private Cancer Genomics Data Visualization and Interpretation
82. A.Krizhevsky I.Sutskever andG. E.Hinton “Imagenet Classification with Deep Convolutional Neural Networks ” inAdvances in Neural Information Processing Systems (NIPS 2012).
83. An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale
84. Improved Denoising Diffusion Probabilistic Models
85. O.Henaff “Data‐Efficient Image Recognition with Contrastive Predictive Coding ” inInternational Conference on Machine Learning (PMLR 2020).
86. P.Bachman R. D.Hjelm andW.Buchwalter “Learning Representations by Maximizing Mutual Information across Views ” inAdvances in Neural Information Processing Systems 30 (NIPS 2019).
87. M.Heusel H.Ramsauer T.Unterthiner B.Nessler andS.Hochreiter “Gans Trained by a Two Time‐scale Update Rule Converge to a Local Nash Equilibrium ”Advances in Neural Information Processing Systems 30 (NIPS 2017).
88. 10.1109/TIP.2003.81986110.1109/TIP.2003.819861 ↗
89. 10.7717/peerj.45310.7717/peerj.453 ↗
90. C.Chen M. Y.Lu D. F. K.Williamson T. Y.Chen A. J.Schaumberg andF.Mahmood “Fast and Scalable Image Search for Histology ”arXiv preprint(2021): 2107.13587.
91. 10.1371/journal.pmed.100273010.1371/journal.pmed.1002730 ↗
92. Quantification of Histochemical Staining by Color Deconvolution
93. Scikit‐Learn: Machine Learning in Python
94. K.He X.Zhang S.Ren andJ.Sun “Deep Residual Learning for Image Recognition ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE 2016):770–778 https://doi.org/10.1109/CVPR.2016.90.10.1109/CVPR.2016.90 ↗
95. 10.1038/s41592‐019‐0686‐210.1038/s41592‐019‐0686‐2 ↗