ColorMedGAN: A Semantic Colorization Framework for Medical Images
Source abstract
Abstract
Colorization for medical images helps make medical visualizations more engaging, provides better visualization in 3D reconstruction, acts as an image enhancement technique for tasks such as segmentation, and makes it easier for non-specialists to perceive tissue changes and texture details in medical images in diagnosis and teaching. However, colorization algorithms have been hindered by limited semantic understanding. In addition, current colorization methods still rely on paired data, which is often not available for specific fields such as medical imaging. To address the texture detail of medical images and the scarcity of paired data, we propose a self-supervised colorization framework based on CycleGAN(Cycle-Consistent Generative Adversarial Networks), treating the colorization problem of medical images as a cross-modal domain transfer problem in color space. The proposed framework focuses on global edge features and semantic information by introducing edge-aware detectors, multi-modal discriminators, and a semantic feature fusion module. Experimental results demonstrate that our method can generate high-quality color medical images.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Image Enhancement Techniques · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information Science and Technology, Jinan University, Guangzhou 510632, China
Guangdong Provincial Key Laboratory of Traditional Chinese Medicine Informatization, Jinan University, Guangzhou 510632, China
College of Cyber Security, Jinan University, Guangzhou 511436, China
School of Economics, Jinan University, Guangzhou 510632, China
International School, Jinan University, Guangzhou 511436, China
Funding
National Key R&D Program of China · Guangdong Basic and Applied Basic Research Poundation · GuangdongProvince Big Data lnnovation Engineering Technology Research Center · “OutstandingPuture” Data Scientist Incubation Project of Jinan University · GuangdongProvincial Key Laboratory of Traditional Chinese Medicine lnformatization
References (52)
Show source-provided references
1. A review of image and video colorization: From analogies to deep learning10.1016/j.visinf.2022.05.003 ↗
2. Zhang, Z., Li, Y., and Shin, B.S. (2022). Robust Medical Image Colorization with Spatial Mask-Guided Generative Adversarial Network. Bioengineering, 9.10.3390/bioengineering9120721 ↗
3. Viana, M.S., Morandin Junior, O., and Contreras, R.C. (2020). An improved local search genetic algorithm with a new mapped adaptive operator applied to pseudo-coloring problem. Symmetry, 12.10.3390/sym12101684 ↗
4. Color Doppler ultrasound imaging of theeye and orbit10.1016/S0039-6257(96)82001-7 ↗
5. Khan, M.U.G., Gotoh, Y., and Nida, N. (2017, January 11–13). Medical image colorization for better visualization and segmentation. Proceedings of the Annual Conference on Medical Image Understanding and Analysis, Edinburgh, UK.10.1007/978-3-319-60964-5_50 ↗
6. Luo, W., Lu, Z., Wang, X., Xu, Y.Q., Ben-Ezra, M., Tang, X., and Brown, M.S. (2012, January 16–21). Synthesizing oil painting surface geometry from a single photograph. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
7. Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., and Brendel, W. (2018). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv.
8. Color transfer between images10.1109/38.946629 ↗
9. Horiuchi, T., and Hirano, S. (2003, January 14–18). Colorization algorithm for grayscale image by propagating seed pixels. Proceedings of the 2003 International Conference on Image Processing (Cat. No. 03CH37429), Barcelona, Spain.
10. A framework for interactive image color editing10.1007/s00371-012-0761-5 ↗
11. Zhang, R., Isola, P., and Efros, A.A. (2016, January 11–14). Colorful image colorization. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_40 ↗
12. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
13. Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
14. Frans, K. (2017). Outline colorization through tandem adversarial networks. arXiv.
15. Generative adversarial networks: An overview10.1109/MSP.2017.2765202 ↗
16. Nazeri, K., Ng, E., and Ebrahimi, M. (2018, January 12–13). Image colorization using generative adversarial networks. Proceedings of the International Conference on Articulated Motion and Deformable Objects, Palma de Mallorca, Spain.10.1007/978-3-319-94544-6_9 ↗
17. Improving CNN linear layers with power mean non-linearity10.1016/j.patcog.2018.12.029 ↗
18. Yi, Z., Zhang, H., Tan, P., and Gong, M. (2017, January 22–29). Dualgan: Unsupervised dual learning for image-to-image translation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.310 ↗
19. Kim, T., Cha, M., Kim, H., Lee, J.K., and Kim, J. (2017, January 6–11). Learning to discover cross-domain relations with generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, NSW, Australia.
20. Kumar, M., Weissenborn, D., and Kalchbrenner, N. (2021). Colorization transformer. arXiv.
21. Reference-based deep line art video colorization
22. Siyao, L., Zhao, S., Yu, W., Sun, W., Metaxas, D., Loy, C.C., and Liu, Z. (2021, January 20–25). Deep animation video interpolation in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00652 ↗
23. Exploring local detail perception for scene sketch semantic segmentation10.1109/TIP.2022.3142511 ↗
24. State of the art in transfer functions for direct volume rendering10.1111/cgf.12934 ↗
25. Adaptive medical image deep color perception algorithm10.1109/ACCESS.2020.2982187 ↗
26. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
27. Medical image coloring based on gabor filtering for internet of medical things10.1109/ACCESS.2020.2999454 ↗
28. Mathur, A.N., Khattar, A., and Sharma, O. (2021, January 19–25). 2D to 3D Medical Image Colorization. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikola, HI, USA.10.1109/WACV48630.2021.00289 ↗
29. Hammami, M., Friboulet, D., and Kéchichian, R. (2020, January 25–28). Cycle GAN-based data augmentation for multi-organ detection in CT images via YOLO. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP40778.2020.9191127 ↗
30. Ma, Y., Liu, Y., Cheng, J., Zheng, Y., Ghahremani, M., Chen, H., Liu, J., and Zhao, Y. (2020, January 4–8). Cycle structure and illumination constrained GAN for medical image enhancement. Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2020: 23rd International Conference, Lima, Peru.10.1007/978-3-030-59713-9_64 ↗
31. Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S., and Yang, B. (2019, January 2–6). Unsupervised medical image translation using cycle-MedGAN. Proceedings of the 2019 27th European Signal Processing Conference (EUSIPCO), A Coruña, Spain.10.23919/EUSIPCO.2019.8902799 ↗
32. Cohen, J.P., Luck, M., and Honari, S. (2018, January 16–20). Distribution matching losses can hallucinate features in medical image translation. Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada, Spain.10.1007/978-3-030-00928-1_60 ↗
33. Breaking the dilemma of medical image-to-image translation
34. Cao, B., Zhang, H., Wang, N., Gao, X., and Shen, D. (2020, January 7–12). Auto-GAN: Self-supervised collaborative learning for medical image synthesis. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6619 ↗
35. A weighted feature transfer gan for medical image synthesis10.1007/s00138-020-01152-8 ↗
36. An adversarial learning approach to medical image synthesis for lesion detection10.1109/JBHI.2020.2964016 ↗
37. Xiao, Y., Jiang, A., Liu, C., and Wang, M. (2019, January 22–25). Single image colorization via modified CycleGAN. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803677 ↗
38. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)10.1109/TMI.2014.2377694 ↗
39. The Visible Human Dataset: The anatomical platform for human simulation10.1002/(SICI)1097-0185(199804)253:2<49::AID-AR8>3.0.CO;2-9 ↗
40. Visible Korean Human: Its techniques and applications
41. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
42. Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2016). Instance normalization: The missing ingredient for fast stylization. arXiv.
43. Li, C., and Wand, M. (2016, January 11–14). Precomputed real-time texture synthesis with markovian generative adversarial networks. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_43 ↗
44. Tang, H., Qi, X., Xu, D., Torr, P.H., and Sebe, N. (2020). Edge guided GANs with semantic preserving for semantic image synthesis. arXiv.10.1145/3394171.3416270 ↗
45. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
46. Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., and Agrawal, A. (2018, January 18–23). Context encoding for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00747 ↗
47. U2-Net: Going deeper with nested U-structure for salient object detection10.1016/j.patcog.2020.107404 ↗
48. Multi-modal volume registration by maximization of mutual information10.1016/S1361-8415(01)80004-9 ↗
49. Gans trained by a two time-scale update rule converge to a local nash equilibrium
50. Seitzer, M. (2021, June 15). Pytorch-Fid: FID Score for PyTorch; Version 0.2.1. Available online: https://github.com/mseitzer/pytorch-fid.
51. Nickparvar, M. (2021, August 20). Brain Tumor MRI Dataset. Available online: https://www.kaggle.com/datasets/navoneel/brain-mri-images-for-brain-tumor-detection.
52. Vitoria, P., Raad, L., and Ballester, C. (2020, January 1–5). Chromagan: Adversarial picture colorization with semantic class distribution. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.10.1109/WACV45572.2020.9093389 ↗