ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographies
Source abstract
Abstract
While traditional self-supervised learning methods improve performance and robustness across various medical tasks, they rely on single-vector embeddings that may not capture fine-grained concepts such as anatomical structures or organs. The ability to identify such concepts and their characteristics without supervision has the potential to improve pre-training methods, and enable novel applications such as fine-grained image retrieval and concept-based outlier detection. In this paper, we introduce ConceptVAE, a novel pre-training framework that detects and disentangles fine-grained concepts from their style characteristics in a self-supervised manner. We present a suite of loss terms and model architecture primitives designed to discretise input data into a preset number of concepts along with their local style. We validate ConceptVAE both qualitatively and quantitatively, demonstrating its ability to detect fine-grained anatomical structures such as blood pools and septum walls from 2D cardiac echocardiographies. Quantitatively, ConceptVAE outperforms traditional self-supervised methods in tasks such as region-based instance retrieval, semantic segmentation, out-of-distribution detection, and object detection. Additionally, we explore the generation of in-distribution synthetic data that maintains the same concepts as the training data but with distinct styles, highlighting its potential for more calibrated data generation. Overall, our study introduces and validates a promising new pre-training technique based on concept-style disentanglement, opening multiple avenues for developing models for medical image analysis that are more interpretable and explainable than black-box approaches.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning in Healthcare · Domain Adaptation and Few-Shot Learning · COVID-19 diagnosis using AI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Foundational Technologies, Siemens SRL, 500097 Brasov, Romania
Siemens Healthineers, Princeton, NJ 08540, USA
Funding
Siemens Healthineers
References (49)
Show source-provided references
1. 3d self-supervised methods for medical imaging
2. Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., and Chen, T. (2021, January 11–17). Big self-supervised models advance medical image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00346 ↗
3. Huang, Z., Jiang, R., Aeron, S., and Hughes, M.C. (2024, January 16–22). Systematic comparison of semi-supervised and self-supervised learning for medical image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02103 ↗
4. Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., and Tian, Y. (2023). A Cookbook of Self-Supervised Learning. arXiv, Available online: https://arxiv.org/abs/2304.12210.
5. Cabannes, V., Kiani, B., Balestriero, R., LeCun, Y., and Bietti, A. (2023, January 23–29). The ssl interplay: Augmentations, inductive bias, and generalization. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA.
6. Wu, L., Zhuang, J., and Chen, H. (2024, January 16–22). Voco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02158 ↗
7. Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., and Auli, M. (2022, January 25–27). Data2vec: A general framework for self-supervised learning in speech, vision and language. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
8. Wang, Y., Li, Z., Mei, J., Wei, Z., Liu, L., Wang, C., Sang, S., Yuille, A.L., Xie, C., and Zhou, Y. (2023, January 8–12). Swinmm: Masked multi-view with swin transformers for 3d medical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Vancouver, BA, Canada.10.1007/978-3-031-43898-1_47 ↗
9. Liu, P., Zhang, J., Wu, X., Liu, S., Wang, Y., Feng, L., Diao, Y., Liu, Z., Lyu, G., and Chen, Y. (2024). Benchmarking Supervised and Self-Supervised Learning Methods in A Large Ultrasound Multi-task Images Dataset. IEEE J. Biomed. Health Inform., Early Access.10.1109/JBHI.2024.3382604 ↗
10. Efficient deep learning-based automated diagnosis from echocardiography with contrastive self-supervised learning10.1038/s43856-024-00538-3 ↗
11. InsCLR: Improving Instance Retrieval with Self-Supervision
12. Deep Learning for Instance Retrieval: A Survey10.1109/TPAMI.2022.3218591 ↗
13. Understanding Human Object Vision: A Picture Is Worth a Thousand Representations10.1146/annurev-psych-032720-041031 ↗
14. How does the brain solve visual object recognition?10.1016/j.neuron.2012.01.010 ↗
15. Recent advances in understanding object recognition in the human brain: Deep neural networks, temporal dynamics, and context10.12688/f1000research.22296.1 ↗
16. Dive into the details of self-supervised learning for medical image analysis10.1016/j.media.2023.102879 ↗
17. Eddahmani, I., Pham, C.H., Napoléon, T., Badoc, I., Fouefack, J.R., and El-Bouz, M. (2023). Unsupervised learning of disentangled representation via auto-encoding: A survey. Sensors, 23.10.3390/s23042362 ↗
18. Learning disentangled representations in the imaging domain10.1016/j.media.2022.102516 ↗
19. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, Virtual. ICML’20.
20. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 13–19). Momentum Contrast for Unsupervised Visual Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
21. Bardes, A., Ponce, J., and LeCun, Y. (2022, January 25–29). VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. Proceedings of the 10th International Conference on Learning Representations, Virtual.
22. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2022, January 18–24). Masked Autoencoders Are Scalable Vision Learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01553 ↗
23. Baevski, A., Babu, A., Hsu, W.N., and Auli, M. (2023, January 23–29). Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA. ICML’23.
24. Garrido, Q., Chen, Y., Bardes, A., Najman, L., and LeCun, Y. (2023, January 1–5). On the duality between contrastive and non-contrastive self-supervised learning. Proceedings of the 11th International Conference on Learning Representations, Kigali, Rwanda.
25. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 5). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. Proceedings of the 9th International Conference on Learning Representations, Vienna, Austria.
26. Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021, January 11–17). Emerging Properties in Self-Supervised Vision Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada.10.1109/ICCV48922.2021.00951 ↗
27. Tian, Y., Fan, L., Chen, K., Katabi, D., Krishnan, D., and Isola, P. (2024, January 16–22). Learning vision from models rivals learning vision from data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01504 ↗
28. Li, K., Wang, Z., Cheng, Z., Yu, R., Zhao, Y., Song, G., Liu, C., Yuan, L., and Chen, J. (2023, January 17–24). ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation. Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00692 ↗
29. Wang, X., Chen, H., Tang, S., Wu, Z., and Zhu, W. (2023). Disentangled Representation Learning. arXiv, Available online: https://arxiv.org/abs/2211.11695.
30. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019, January 9–15). Challenging common assumptions in the unsupervised learning of disentangled representations. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
31. ClusterGAN: Latent Space Clustering in Generative Adversarial Networks
32. Ngweta, L., Maity, S., Gittens, A., Sun, Y., and Yurochkin, M. (2023, January 23–29). Simple disentanglement of style and content in visual representations. Proceedings of the 40th International Conferenceon Machine Learning, Honolulu, HI, USA. ICML’23.
33. Razavi, A., van den Oord, A., and Vinyals, O. (2019, January 8–14). Generating diverse high-fidelity images with VQ-VAE-2. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
34. Esser, P., Rombach, R., and Ommer, B. (2021, January 20–25). Taming Transformers for High-Resolution Image Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01268 ↗
35. Disentangled representation learning in cardiac image analysis10.1016/j.media.2019.101535 ↗
36. RetCCL: Clustering-guided contrastive learning for whole-slide image retrieval10.1016/j.media.2022.102645 ↗
37. Self-supervised contrastive learning with random walks for medical image segmentation with limited annotations10.1016/j.compmedimag.2022.102174 ↗
38. Jang, E., Gu, S., and Poole, B. (2017, January 24–26). Categorical Reparameterization with Gumbel-Softmax. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
39. Park, T., Liu, M., Wang, T., and Zhu, J. (2019, January 15–20). Semantic Image Synthesis with Spatially-Adaptive Normalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00244 ↗
40. Self-supervised learning for medical image analysis: Discriminative, restorative, or adversarial?10.1016/j.media.2024.103086 ↗
41. Nekrasov, V., Shen, C., and Reid, I. (2018). Light-weight refinenet for real-time semantic segmentation. arXiv, Available online: https://arxiv.org/abs/1810.03272.
42. Sketch-based semantic retrieval of medical images10.1016/j.media.2023.103060 ↗
43. Large-scale retrieval for medical image analytics: A comprehensive review10.1016/j.media.2017.09.007 ↗
44. Ren, J., Fort, S., Liu, J., Roy, A.G., Padhy, S., and Lakshminarayanan, B. (2021). A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection. arXiv, Available online: https://arxiv.org/abs/2106.09022.
45. Kuan, J., and Mueller, J. (2022). Back to the Basics: Revisiting Out-of-Distribution Detection Baselines. arXiv, Available online: https://arxiv.org/abs/2207.03061.
46. Normalizing Flows: An Introduction and Review of Current Methods10.1109/TPAMI.2020.2992934 ↗
47. van Amersfoort, J., Smith, L., Jesson, A., Key, O., and Gal, Y. (2022). On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty. arXiv, Available online: https://arxiv.org/abs/2102.11409.
48. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
49. Variability of echocardiographic measures of left ventricular diastolic function. The HUNT study10.1111/echo.15073 ↗