A Multi-Task Learning and Multi-Branch Network for DR and DME Joint Grading
Source abstract
Abstract
Diabetic Retinopathy (DR) is one of the most common microvascular complications of diabetes. Diabetic Macular Edema (DME) is a concomitant symptom of DR. As the grade of lesion of DR and DME increase, the possibility of blindness can also increase significantly. In order to take the early interventions as soon as possible to reduce the likelihood of blindness, it is necessary to perform both DR and DME grading. We design a joint grading model based on multi-task learning and multi-branch networks (MaMNet) for DR and DME grading. The model mainly includes a multi-branch network (MbN), a feature fusion module, and a disease classification module. The MbN is formed by four branch structures, which can extract the low-level feature information of DME and DR in a targeted way; the feature fusion module is composed of a self-feature extraction module (SFEN), cross-feature extraction module (CFEN) and atrous spatial pyramid pooling module (ASPP). By combining various features collected from the aforementioned modules, the feature fusion module can provide more thorough discriminative features, which benefits the joint grading accuracy. The ISBI-2018-IDRiD challenge dataset is used to evaluate the performance of the proposed model. The experimental results show that based on the multi-task strategy the two grading tasks of DR and DME can provide each other with additional useful information. The joint accuracy of the model, the accuracy of DR and the accuracy of DME are 61.2%, 64.1% and 79.4% respectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Retinal Imaging and Analysis · Retinal Diseases and Treatments · Digital Imaging for Blood Diseases
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronic Information Engineering, Changchun University, Changchun 130012, China
Image Engineering & Video Technology Lab, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
Funding
Project of Jilin Provincial Department of Science and Technology
References (47)
Show source-provided references
1. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales10.1016/S0161-6420(03)00475-5 ↗
2. Diabetic retinopathy and diabetic macular edema: Pathophysiology, screening, and novel therapies10.2337/diacare.26.9.2653 ↗
3. Applications of deep learning in fundus images: A review10.1016/j.media.2021.101971 ↗
4. Classification of diabetic retinopathy and diabetic macular edema10.4239/wjd.v4.i6.290 ↗
5. Convolutional neural networks for diabetic retinopathy10.1016/j.procs.2016.07.014 ↗
6. Automated identification of diabetic retinopathy using deep learning10.1016/j.ophtha.2017.02.008 ↗
7. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs10.1001/jama.2016.17216 ↗
8. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
9. Automated identification and grading system of diabetic retinopathy using deep neural networks10.1016/j.knosys.2019.03.016 ↗
10. Deep learning-based automated detection for diabetic retinopathy and diabetic macular oedema in retinal fundus photographs10.1038/s41433-021-01552-8 ↗
11. Wang, Z., Yin, Y., Shi, J., Fang, W., Li, H., and Wang, X. (2017, January 11–13). Zoom-in-net: Deep mining lesions for diabetic retinopathy detection. Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2017: 20th International Conference, Quebec City, QC, Canada.10.1007/978-3-319-66179-7_31 ↗
12. Lin, Z., Guo, R., Wang, Y., Wu, B., Chen, T., Wang, W., and Wu, J. (2018, January 16–20). A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion. Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain.10.1007/978-3-030-00934-2_9 ↗
13. Zhou, Y., He, X., Huang, L., Liu, L., Zhu, F., Cui, S., and Shao, L. (2019, January 16–20). Collaborative learning of semi-supervised segmentation and classification for medical images. Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00218 ↗
14. A novel machine learning model based on exudate localization to detect diabetic macular edema
15. Exudate-based diabetic macular edema recognition in retinal images using cascaded deep residual networks10.1016/j.neucom.2018.02.035 ↗
16. He, X., Zhou, Y., Wang, B., Cui, S., and Shao, L. (2019, January 13–17). Dme-net: Diabetic macular edema grading by auxiliary task learning. Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China.10.1007/978-3-030-32239-7_87 ↗
17. Multitask learning10.1023/A:1007379606734 ↗
18. Tan, C., Zhao, L., Yan, Z., Li, K., Metaxas, D., and Zhan, Y. (2018, January 4–7). Deep multi-task and task-specific feature learning network for robust shape preserved organ segmentation. Proceedings of the 15th IEEE International Symposium on Biomedical Imaging, Washington, DC, USA.10.1109/ISBI.2018.8363791 ↗
19. Liu, L., Dou, Q., Chen, H., Olatunji, I.E., Qin, J., and Heng, P.A. (2018, January 20). Mtmr-net: Multi-task deep learning with margin ranking loss for lung nodule analysis. Proceedings of the 4th International Workshop on Deep Learning in Medical Image Analysis (DLMIA)/8th International Workshop on Multimodal Learning for Clinical Decision Support (ML-CDS), Granada, Spain.10.1007/978-3-030-00889-5_9 ↗
20. A multitask deep learning model for the classification of age-related macular degeneration
21. Multiple organ localization in CT image using triple-branch fully convolutional networks10.1109/ACCESS.2019.2930417 ↗
22. A distributed multitask multimodal approach for the prediction of Alzheimer’s disease in a longitudinal study10.1016/j.neuroimage.2019.116317 ↗
23. A multitask, multi-stage deep transfer learning model for early prediction of neurodevelopment in very preterm infants10.1038/s41598-020-71914-x ↗
24. Deep learning-based concurrent brain registration and tumor segmentation10.3389/fncom.2020.00017 ↗
25. Predicting treatment response from longitudinal images using multi-task deep learning10.1038/s41467-021-22188-y ↗
26. Hao, P., Gao, X., Li, Z., Zhang, J., Wu, F., and Bai, C. (2020). Multi-branch fusion network for Myocardial infarction screening from 12-lead ECG images. Comput. Methods Programs Biomed., 184.10.1016/j.cmpb.2019.105286 ↗
27. Zhuang, J. (2018). LadderNet: Multi-path networks based on U-Net for medical image segmentation. arXiv.
28. EMS-Net: Ensemble of multiscale convolutional neural networks for classification of breast cancer histology images10.1016/j.neucom.2019.07.080 ↗
29. An attentive survey of attention models10.1145/3465055 ↗
30. Multi-scale self-guided attention for medical image segmentation10.1109/JBHI.2020.2986926 ↗
31. Cai, Y., and Wang, Y. (2021, January 5). Ma-unet: An improved version of unet based on multi-scale and attention mechanism for medical image segmentation. Proceedings of the Third International Conference on Electronics and Communication; Network and Computer Technology (ECNCT 2021), Xiamen, China.10.1117/12.2628519 ↗
32. Valanarasu, J.M.J., Oza, P., Hacihaliloglu, I., and Patel, V.M. (October, January 27). Medical transformer: Gated axial-attention for medical image segmentation. Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France.10.1007/978-3-030-87193-2_4 ↗
33. Wu, Z., Su, L., and Huang, Q. (2019, January 15–20). Cascaded partial decoder for fast and accurate salient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00403 ↗
34. Cross-attention multi-branch network for fundus diseases classification using SLO images10.1016/j.media.2021.102031 ↗
35. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
36. Peng, C., Zhang, X., Yu, G., Luo, G., and Sun, J. (2017, January 21–26). Large kernel matters--improve semantic segmentation by global convolutional network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.189 ↗
37. Zhao, T., and Wu, X. (2019, January 15–20). Pyramid feature attention network for saliency detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00320 ↗
38. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
39. Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., and Meriaudeau, F. (2018). Indian diabetic retinopathy image dataset (IDRiD): A database for diabetic retinopathy screening research. Data, 3.10.3390/data3030025 ↗
40. (2019, March 27). Diabetic Retinopathy: Segmentation and Grading Challenge. Available online: https://idrid.grand-challenge.org/Leaderboard/.
41. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
42. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
43. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
44. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
45. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
46. Yu, F., Wang, D., Shelhamer, E., and Darrell, T. (2018, January 18–22). Deep layer aggregation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00255 ↗
47. Gene selection with Game Shapley Harris hawks optimizer for cancer classification10.1016/j.chemolab.2023.104989 ↗