Enhanced detection of diabetic retinopathy using machine learning based feature selection and ensemble classifiers

Diabetic Retinopathy (DR) is a progressive eye disorder commonly observed in individuals with long-term diabetes. As the number of people with diabetes is increasing, it is difficult to get the constant attention of ophthalmologists. Automatic detection with accuracy is necessary. This study proposes a novel hybrid classification framework for DR detection, integrating advanced deep learning and machine learning techniques to improve accuracy and reliability. The proposed framework employs a generative adversarial network-based augmentation technique for data augmentation and ResNet101-based transfer learning for feature extraction, and reinforcement learning improves model performance by identifying the most significant features. The hybrid XGBoost-LSTM ensemble classifier is developed to optimize DR classification by leveraging both sequential dependencies and gradient-boosted decision trees. The efficacy of the proposed method is evaluated through the analysis of three standard datasets—Kaggle EyePACS dataset, MESSIDOR, and APTOS—to establish its robustness and generalizability. The proposed method demonstrates enhanced performance over conventional machine learning classifiers, such as extra trees, support vector machines, logistic regression, random forest, and multi-layer perceptron. Furthermore, it outperforms pre-trained deep learning (DL) models such as LeNet-5, VGG16, ResNet50, Inception V3, EfficientNet, DenseNet, and AlexNet. The proposed hybrid model achieves superior performance across all datasets, attaining 98.60% accuracy and 97.46% Area Under the Curve (AUC) on the Kaggle dataset, 98.60% accuracy and 96.75% AUC on the MESSIDOR dataset, and 98.75% accuracy and 96.22% AUC on the APTOS dataset. The comparative analysis underscores the limitations of conventional feature extraction and classification methods, while the proposed method effectively utilizes DL and ensemble techniques to enhance DR detection accuracy and reliability.