Efficient Knowledge Distillation for Brain Tumor Segmentation
Source abstract
Abstract
Deep learning has allowed great progress to be made in obtaining more accurate prediction results for brain tumor segmentation. The current mainstream research approaches obtain segmentation accuracy improvements by modifying deep-learning model architectures, while ignoring the computational and storage efficiency issues of segmentation. In this paper, we proposed an improved knowledge distillation method: coordinate distillation (CD), which integrates channel and space information and completes brain tumor segmentation by training the student network with the teacher network without changing the original network architecture. Experimental results showed that the method was effective and that it could enhance the segmentation accuracy of brain tumors without changing the segmentation efficiency.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Brain Tumor Detection and Classification · Advanced Neural Network Applications · Medical Image Segmentation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Engineering, Linyi University, Linyi 276000, China
College of Engineering, University of Sydney, Sydney 2006, Australia
Funding
National Science Foundation of China (NSFC) · Natural Science Foundation of Shandong Province
References (38)
Show source-provided references
1. A survey on deep learning in medical image analysis10.1016/j.media.2017.07.005 ↗
2. The 2016 World Health Organization classification of tumors of the central nervous system: A summary10.1007/s00401-016-1545-1 ↗
3. Liu, Z., Tong, L., Chen, L., Jiang, Z., Zhou, F., Zhang, Q., Zhang, X., Jin, Y., and Zhou, H. (2022). Deep learning based brain tumor segmentation: A survey. Complex Intell. Syst., 1–26.10.1007/s40747-022-00815-5 ↗
4. A mutual bootstrapping model for automated skin lesion segmentation and classification10.1109/TMI.2020.2972964 ↗
5. SESV: Accurate medical image segmentation by predicting and correcting errors10.1109/TMI.2020.3025308 ↗
6. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
7. Training deep neural networks in generations: A more tolerant teacher educates better students
8. Xie, Q., Luong, M.T., Hovy, E., and Le, Q.V. (2020, January 13–19). Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01070 ↗
9. Phuong, M., and Lampert, C.H. (2019, January 27–28). Distillation-based training for multi-exit architectures. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00144 ↗
10. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv.
11. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). Tinybert: Distilling bert for natural language understanding. arXiv.10.18653/v1/2020.findings-emnlp.372 ↗
12. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
13. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers
14. Lachinov, D., Shipunova, E., and Turlapov, V. (2019). Knowledge distillation for brain tumor segmentation. International MICCAI Brainlesion Workshop, Springer.10.1007/978-3-030-46643-5_32 ↗
15. Hu, M., Maillard, M., Zhang, Y., Ciceri, T., La Barbera, G., Bloch, I., and Gori, P. (2020). Knowledge distillation from multi-modal to mono-modal segmentation networks. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-030-59710-8_75 ↗
16. Nalwade, A., and Kisa, J. (2021). Experimenting with Knowledge Distillation techniques for performing Brain Tumor Segmentation. arXiv.
17. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
18. Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., and Maier-Hein, K.H. (2017). Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge. International MICCAI Brainlesion Workshop, Springer.10.1007/978-3-319-75238-9_25 ↗
19. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation10.1109/TMI.2019.2959609 ↗
20. Road extraction by deep residual u-net10.1109/LGRS.2018.2802944 ↗
21. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018). Attention u-net: Learning where to look for the pancreas. arXiv.
22. Chen, Y., Cao, Z., Cao, C., Yang, J., and Zhang, J. (2018). A modified U-Net for brain Mr image segmentation. International Conference on Cloud Computing and Security, Springer.10.1007/978-3-030-00021-9_22 ↗
23. Deng, Y., Sun, Y., Zhu, Y., Zhu, M., Han, W., and Yuan, K. (2018). A strategy of MR brain tissue images’ suggestive annotation based on modified U-net. arXiv.
24. Kong, X., Sun, G., Wu, Q., Liu, J., and Lin, F. (2018). Hybrid pyramid u-net model for brain tumor segmentation. International Conference on Intelligent Information Processing, Springer.10.1007/978-3-030-00828-4_35 ↗
25. Milletari, F., Navab, N., and Ahmadi, S.A. (2016, January 25–28). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
26. Inter-slice context residual learning for 3D medical image segmentation10.1109/TMI.2020.3034995 ↗
27. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., and Ronneberger, O. (2016). 3D U-Net: Learning dense volumetric segmentation from sparse annotation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-46723-8_49 ↗
28. Knowledge distillation: A survey10.1007/s11263-021-01453-z ↗
29. Zhang, Y., Xiang, T., Hospedales, T.M., and Lu, H. (2018, January 18–22). Deep mutual learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00454 ↗
30. Cho, J.H., and Hariharan, B. (2019, January 27–28). On the efficacy of knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00489 ↗
31. Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., and Wang, J. (2019, January 16–17). Structured knowledge distillation for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00271 ↗
32. Efficient medical image segmentation based on knowledge distillation10.1109/TMI.2021.3098703 ↗
33. Zagoruyko, S., and Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv.
34. Yim, J., Joo, D., Bae, J., and Kim, J. (2017, January 21–26). A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.754 ↗
35. Tung, F., and Mori, G. (2019, January 27–28). Similarity-preserving knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00145 ↗
36. On information and sufficiency10.1214/aoms/1177729694 ↗
37. Hou, Q., Zhou, D., and Feng, J. (2021, January 19–25). Coordinate attention for efficient mobile network design. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.01350 ↗
38. The multimodal brain tumor image segmentation benchmark (BRATS)10.1109/TMI.2014.2377694 ↗