Multi-Task Mean Teacher Medical Image Segmentation Based on Swin Transformer
Source abstract
Abstract
As a crucial task for disease diagnosis, existing semi-supervised segmentation approaches process labeled and unlabeled data separately, ignoring the relationships between them, thereby limiting further performance improvements. In this work, we introduce a transformer-based multi-task framework that concurrently leverages both labeled and unlabeled volumes by encoding shared representation patterns. We first integrate transformers into YOLOv5 to enhance segmentation capabilities and adopt a multi-task approach spanning shadow region detection and boundary localization. Subsequently, we leverage the mean teacher model to simultaneously learn from labeled and unlabeled inputs alongside orthogonal view representations, enabling our approach to harness all available annotations. Our network can improve the learning ability and attain superior performance. Extensive experiments demonstrate that the transformer-powered architecture encodes robust inter-sample relationships, unlocking substantial performance gains by capturing shared information between labeled and unlabeled data. By treating both data types concurrently and encoding their shared patterns, our framework addresses the limitations of existing semi-supervised approaches, leading to improved segmentation accuracy and robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Image and Object Detection Techniques · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Xi’an University of Technology, Xi’an 710048, China
Shaanxi Province Key Laboratory of Network Computing and Security Technology, Xi’an University of Technology, Xi’an 710048, China
Funding
NSFC
References (52)
Show source-provided references
1. Chest CT manifestations of new coronavirus disease 2019 (COVID-19): A pictorial review10.1007/s00330-020-06801-0 ↗
2. Milletari, F., Navab, N., and Ahmadi, S.-A. (2016, January 25–28). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
3. Medical image segmentation using deep learning: A survey10.1049/ipr2.12419 ↗
4. Jiao, R., Zhang, Y., Ding, L., Xue, B., Zhang, J., Cai, R., and Jin, C. (2023). Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation. Comput. Biol. Med., 169.10.1016/j.compbiomed.2023.107840 ↗
5. Deep learning techniques for automatic mri cardiac multistructures segmentation and diagnosis: Is the problem solved?10.1109/TMI.2018.2837502 ↗
6. Deep learning methods for automatic evaluation of delayed enhancement-mri. the results of the emidec challenge10.1016/j.media.2022.102428 ↗
7. A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging10.1016/j.media.2020.101832 ↗
8. Abdomenct-1k: Is abdominal organ segmentation a solved problem10.1109/TPAMI.2021.3100536 ↗
9. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge10.1016/j.media.2020.101821 ↗
10. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation10.1016/j.media.2020.101693 ↗
11. Exploiting shared knowledge from non-covid lesions for annotation-effificient COVID-19 ct lung infection segmentation10.1109/JBHI.2021.3106341 ↗
12. Chen, Z., Zhu, L., Wan, L., Wang, S., Feng, W., and Heng, P.-A. (2020, January 13–19). A Multi-task Mean Teacher for Semi-supervised Shadow Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020), Seattle, WA, USA.10.1109/CVPR42600.2020.00565 ↗
13. You, C., Dai, W., Min, Y., Liu, F., Clifton, D.A., Zhou, S.K., Staib, L.H., and Duncan, J.S. (2024, January 10–16). Rethinking semi-supervised medical image segmentation: A variance-reduction perspective. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA.
14. Tarvainen, A., and Valpola, H. (2017, January 4–9). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
15. Li, S., Zhang, C., and He, X. (2020, January 4–8). Shape-aware semi-supervised 3d semantic segmentation for medical images. Proceedings of the Conference on Medical Image Computing and Computer—Assisted Intervention, Lima, Peru.10.1007/978-3-030-59710-8_54 ↗
16. Semi-supervised medical image segmentation through dual-task consistency
17. Wu, J., Fu, R.A., Fang, H., Zhang, Y., Yang, Y., Xiong, H., Liu, H., and Xu, Y. (2024, January 9–11). Medsegdiff: Medical image segmentation with diffusion probabilistic model. Proceedings of the Medical Imaging with Deep Learning, Tromsø, Norway.
18. Attention gated networks: Learning to leverage salient regions in medical images10.1016/j.media.2019.01.012 ↗
19. Zhang, J., Xiang, K., Wang, J., Liu, J., Kang, M., and Pan, Z. (2022, January 26–28). Trans-Inf-Net: COVID-19 Lung Infection Segmentation based on Transformer. Proceedings of the 8th ICVR, Nanjing, China.10.1109/ICVR55215.2022.9848070 ↗
20. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., and Wang, M. (2022, January 23–27). Swin-unet: Unet-like pure transformer for medical image segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-25066-8_9 ↗
21. Review of artifificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-1910.1109/RBME.2020.2987975 ↗
22. Multiple resolution residually connected feature streams for automatic lung tumor segmentation from CT images10.1109/TMI.2018.2857800 ↗
23. Zhang, Y., Zhou, Z., David, P., Yue, X., Xi, Z., Gong, B., and Foroosh, H. (2020, January 13–19). PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00962 ↗
24. Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J.R., MaierHein, K., and Ronneberger, O. (2018, January 2–8). A probabilistic u-net for segmentation of ambiguous images. Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada.
25. Ronneberger, O., Fischer, P., and Brox, T. (2015). Medical Image Computing and Computer-Assisted Intervention—MICCAI, Springer.
26. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Glocker, B., and Rueckert, D. (2018). Attention u-net: Learning where to look for the pancreas. arXiv.
27. Zhao, N., Tong, N., Ruan, D., and Sheng, K. (2019). Medical Image Computing and Computer Assisted Intervention—MICCAI, Springer.
28. Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., and Chandraker, M. (2018, January 18–23). Learning to adapt structured output space for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00780 ↗
29. Tang, Y., Tang, Y., Xiao, J., and Summers, R.M. (2019, January 13–17). TUNA-Net: Task-oriented Unsupervised Adversarial Network for Disease Recognition in CrossDomain Chest X-rays. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32226-7_48 ↗
30. Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis10.1016/j.media.2019.03.009 ↗
31. Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., and Chen, D.Z. (2017). Medical Image Computing and Computer-Assisted Intervention—MICCAI, Springer.
32. Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P.M., and Rueckert, D. (2017). Medical Image Computing and Computer-Assisted Intervention—MICCAI, Springer.
33. Krähenbühl, P., and Koltun, V. (2011, January 2–14). Effificient inference in fully connected crfs with gaussian edge potentials. Proceedings of the 25th Annual Conference on Neural Information Processing Systems 2011, Granada, Spain.
34. Yu, L., Wang, S., Li, X., Fu, C.-W., and Heng, P.-A. (2019). Medical Image Computing and Computer-Assisted Intervention—MICCAI, Springer.
35. Roy, S., Koehler, G., Ulrich, C., Baumgartner, M., Petersen, J., Isensee, F., Jaeger, P.F., and Maier-Hein, K. (2023, January 8–12). Mednext: Transformer-driven scaling of convnets for medical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Vancouver, BC, Canada.10.1007/978-3-031-43901-8_39 ↗
36. Rahman, M.M., and Marculescu, R. (2024, January 3–5). Multi-scale hierarchical vision transformer with cascaded attention decoding for medical image segmentation. Proceedings of the Medical Imaging with Deep Learning, Paris, France.10.1109/WACV56688.2023.00616 ↗
37. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018, January 10–15). Image transformer. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
38. Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv.
39. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 4). An image is worth 16 × 16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
40. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
41. Xie, S., Girshick, R., Doll, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
42. Deeply supervised salient object detection with short connections10.1109/TPAMI.2018.2815688 ↗
43. Zhu, L., Deng, Z., Hu, X., Fu, C.-W., Xu, X., Qin, J., and Heng, P.-A. (2018, January 8–14). Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_8 ↗
44. Fan, D.-P., Yang, J.-F., Cheng, M.-M., Zhao, J.-X., and Liu, J.-J. (November, January 27). EGNet: Edge guidance network for salient object detection. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
45. Cheng, M.-M., Feng, J., Jiang, J., Liu, J., and Hou, Q. (2019, January 15–20). A simple pooling-based design for real-time salient object detection. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.
46. Chen, S., Tan, X., Wang, B., and Hu, X. (2018, January 8–14). Reverse attention for salient object detection. Proceedings of the ECCV 2018 European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01240-3_15 ↗
47. A computational approach to edge detection10.1109/TPAMI.1986.4767851 ↗
48. Zhang, Y., and Zhang, J. (November, January 29). Dual-task mutual learning for semi-supervised medical image segmentation. Proceedings of the Pattern Recognition and Computer Vision: 4th Chinese Conference, PRCV 2021, Beijing, China.10.1007/978-3-030-88010-1_46 ↗
49. Yang, X., He, X., Zhao, J., Zhang, Y., Zhang, S., and Xie, P. (2020). COVID-CT-dataset: A CT scan dataset about COVID-19. arXiv.
50. Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images10.1109/TMI.2020.2996645 ↗
51. Fan, D.-P., Cheng, M.-M., Liu, Y., Li, T., and Borji, A. (2017, January 22–29). Structuremeasure: A new way to evaluate foreground maps. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.487 ↗
52. Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., and Borji, A. (2018, January 13–19). Enhanced-alignment measure for binary foreground map evaluation. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/97 ↗