A Multimodal Dual-Stream Cross-Attention Deep Learning Framework for Diabetic Foot Ulcer Classification

Finding diabetic foot ulcers (DFUs) early and accurately is essential for improving patients’ quality of life and lowering the risk of amputation. RGB images, commonly used in automated DFU detection, have limitations such as lighting variations, color inconsistencies, and inability to directly reflect physiological information. Background/Objectives: Although thermal images can capture temperature anomalies associated with inflammation and circulatory disorders, they cannot provide consistent performance due to their low spatial resolution and limited availability in clinical datasets. Furthermore, the lack of paired RGB–thermal image pairs makes it difficult to develop effective multimodal deep learning models. Methods: This study proposes a two-stage multimodal deep learning approach to overcome these limitations. In the first stage, an RGB2T-cGAN (RGB to Thermal cGAN) model based on pix2pix was designed to generate synthetic thermal representations from RGB images that resemble clinical patterns, thereby addressing the missing modality problem. In the second stage, the Multimodal Dual-Stream Multi-Head Cross-Attention (MDS-MHCA) classifier model was developed, which processes DFU RGB and generated synthetic thermal images through separate streams, enabling the dynamic modeling of complementary information across modalities. Results: The proposed MDS-MHCA model achieved 99.06% accuracy, 99.09% recall, and 99.06% F1-score on the test set, demonstrating a clear advantage over models based solely on RGB (91.51% accuracy) or thermal (96.23% accuracy) modalities. Furthermore, patient-based 10-fold GroupKFold cross-validation results demonstrate that the model offers high generalization capability across different patient groups, with an average accuracy of 96.49 ± 1.04 and an AUC value of 0.9927 ± 0.0067. Conclusions: The findings reveal that the proposed approach, through the integration of synthetic thermal information and cross-attention-based multimodal fusion, overcomes the fundamental limitations of single-modality-based systems and offers a DFU detection system that is more robust and reliable and holds potential for integration into clinical decision support systems.