journal article·2025-09-21Open access

A Multimodal Deep Learning Framework for Accurate Wildfire Segmentation Using RGB and Thermal Imagery

Tao Yue, Hong Huang, Qingyang Wang, Bo Song, Yun Chen

Applied Sciences· Volume 15· Issue 18

Abstract

Wildfires pose serious threats to ecosystems, human life, and climate stability, underscoring the urgent need for accurate monitoring. Traditional approaches based on either optical or thermal imagery often fail under challenging conditions such as lighting interference, varying data sources, or small-scale flames, as they do not account for the hierarchical nature of feature representations. To overcome these limitations, we propose a multimodal deep learning framework that integrates visible (RGB) and thermal infrared (TIR) imagery for accurate wildfire segmentation. The framework incorporates edge-guided supervision and multilevel fusion to capture fine fire boundaries while exploiting complementary information from both modalities. To assess its effectiveness, we constructed a multi-scale flame segmentation dataset and validated the method across diverse conditions, including different data sources, lighting environments, and five flame size categories ranging from small to large. Experimental results show that BFCNet achieves an IoU of 88.25% and an F1 score of 93.76%, outperforming both single-modality and existing multimodal approaches across all evaluation tasks. These results demonstrate the potential of multimodal deep learning to enhance wildfire monitoring, offering practical value for disaster management, ecological protection, and the deployment of autonomous aerial surveillance systems.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Fire Detection and Safety Systems · Fire effects on ecosystems · Video Surveillance and Tracking Methods

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

College of Geomatics and Geoinformation, Guilin University of Technology, Guilin 541004, China

College of Environmental Science and Engineering, Guilin University of Technology, Guilin 541006, China

Guangxi Key Laboratory of Environmental Pollution Control Theory and Technology, Guilin University of Technology, Guilin 541006, China

Key Laboratory of Carbon Emission and Pollutant Collaborative Control of Education, Department of Guangxi Zhuang Autonomous Region, Guilin University of Technology, Guilin 541006, China

Funding

Guangxi Science and Technology Project · National Natural Science Foundation of China

References (49)

Show source-provided references
1. Akhloufi, M.A., Couturier, A., and Castro, N.A. (2021). Unmanned aerial vehicles for wildland fires: Sensing, perception, cooperation and assistance. Drones, 5.10.3390/drones5010015 ↗
2. FlameFinder: Illuminating Obscured Fire Through Smoke with Attentive Deep Metric Learning10.1109/TGRS.2024.3440880 ↗
3. A satellite imagery smoke detection framework based on the Mahalanobis distance for early fire identification and positioning
4. Domain-incremental learning for fire detection in space-air-ground integrated observation network
5. Aerial imagery pile burn detection using deep learning: The FLAME dataset10.1016/j.comnet.2021.108001 ↗
6. Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection10.1109/TCSVT.2014.2339592 ↗
7. Higher order linear dynamical systems for smoke detection in video surveillance applications10.1109/TCSVT.2016.2527340 ↗
8. Classification of multidimensional time-evolving data using histograms of grassmannian points10.1109/TCSVT.2016.2631719 ↗
9. Multi-feature fusion based fast video flame detection10.1016/j.buildenv.2009.10.017 ↗
10. Optical flow estimation for flame detection in videos10.1109/TIP.2013.2258353 ↗
11. An improved probabilistic approach for fire detection in videos10.1007/s10694-012-0253-1 ↗
12. An image processing technique for fire detection in video images10.1016/j.firesaf.2006.02.001 ↗
13. Computer vision based method for real-time fire and flame detection10.1016/j.patrec.2005.06.015 ↗
14. Fire detection in video sequences using a generic color model10.1016/j.firesaf.2008.05.005 ↗
15. Efficient flame detection based on static and dynamic texture analysis in forest fire detection10.1007/s10694-017-0683-x ↗
16. Computer vision for wildfire research: An evolving image dataset for processing and analysis10.1016/j.firesaf.2017.06.012 ↗
17. Deep learning in neural networks: An overview10.1016/j.neunet.2014.09.003 ↗
18. A review on early wildfire detection from unmanned aerial vehicles using deep learning-based computer vision algorithms10.1016/j.sigpro.2021.108309 ↗
19. Barmpoutis, P., Papaioannou, P., Dimitropoulos, K., and Grammalidis, N. (2020). A review on early forest fire detection systems using optical remote sensing. Sensors, 20.10.3390/s20226442 ↗
20. Fire detection using autonomous aerial vehicles with infrared and visual cameras10.3182/20050703-6-CZ-1902.01380 ↗
21. Rossi, L., Toulouse, T., Akhloufi, M., Pieri, A., and Tison, Y. (2013, January 12). Estimation of spreading fire geometrical characteristics using near infrared stereovision. Proceedings of the Three-Dimensional Image Processing (3DIP) and Applications, Burlingame, CA, USA.10.1117/12.2001624 ↗
22. Lu, Y., Wu, Y., Liu, B., Zhang, T., Li, B., Chu, Q., and Yu, N. (2020, January 13–19). Cross-modality person reidentification with shared-specific feature transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01339 ↗
23. Zhao, Y., Ma, J., Li, X., and Zhang, J. (2018). Saliency detection and deep learning-based wildfire identification in UAV imagery. Sensors, 18.10.3390/s18030712 ↗
24. Barmpoutis, P., Stathaki, T., Dimitropoulos, K., and Grammalidis, N. (2020). Early fire detection based on aerial 360-degree sensors, deep convolution neural networks and exploitation of fire dynamic textures. Remote Sens., 12.10.3390/rs12193177 ↗
25. CNN-based, contextualized, real-time fire detection in computational resource-constrained environments10.1016/j.egyr.2023.05.260 ↗
26. Shirvani, Z., Abdi, O., and Goodman, R.C. (2023). High-resolution semantic segmentation of woodland fires using residual attention UNet and time series of Sentinel-2. Remote Sens., 15.10.3390/rs15051342 ↗
27. Active fire detection in Landsat-8 imagery: A large-scale dataset and a deep-learning study10.1016/j.isprsjprs.2021.06.002 ↗
28. Hu, X., Jiang, F., Qin, X., Huang, S., Yang, X., and Meng, F. (2024). An optimized smoke segmentation method for forest and grassland fire based on the UNet framework. Fire, 7.10.3390/fire7030068 ↗
29. Wildfire Detection Powered by Involutional Neural Network and Multi-Task Learning with Dark Channel Prior Technique10.1109/JSTARS.2024.3450714 ↗
30. Zhou, T., Fu, H., Chen, G., Zhou, Y., Fan, D.-P., and Shao, L. (2021, January 20–25). Specificity-preserving RGB-D saliency detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), Nashville, TN, USA.10.1109/ICCV48922.2021.00464 ↗
31. A RGB-Thermal based adaptive modality learning network for day–night wildfire identification
32. BA_EnCaps: Dense capsule architecture for thermal scrutiny10.1109/TGRS.2022.3166352 ↗
33. Improving RGB-Thermal Semantic Scene Understanding with Synthetic Data Augmentation for Autonomous Driving10.1109/LRA.2025.3548399 ↗
34. PEAFusion: Parameter-efficient Adaptation for RGB-Thermal fusion-based semantic segmentation10.1016/j.inffus.2025.103030 ↗
35. M2FNet: Mask-Guided Multi-Level Fusion for RGB-T Pedestrian Detection10.1109/TMM.2024.3381377 ↗
36. Dual-stream siamese network for RGB-T dual-modal fusion object tracking on UAV10.1007/s11227-025-07519-w ↗
37. RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes10.1109/LRA.2019.2904733 ↗
38. Deng, F., Feng, H., Liang, M., Wang, H., Yang, Y., Gao, Y., Chen, J., Hu, J., Guo, X., and Lam, T.L. (October, January 27). FEANet: Feature-enhanced attention network for RGB-thermal real-time semantic segmentation. Proceedings of the 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic.10.1109/IROS51168.2021.9636084 ↗
39. GMNet: Graded-feature multilabel-learning network for RGB-thermal urban scene semantic segmentation10.1109/TIP.2021.3109518 ↗
40. EGFNet: Edge-aware guidance fusion network for RGB–thermal urban scene parsing10.1109/TITS.2023.3306368 ↗
41. Guo, S., Hu, B., and Huang, R. (2021, January 27–31). Real-time flame segmentation based on rgb-thermal fusion. Proceedings of the 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), Sanya, China.10.1109/ROBIO54168.2021.9739283 ↗
42. Wildland fire detection and monitoring using a drone-collected rgb/ir image dataset10.1109/ACCESS.2022.3222805 ↗
43. Early wildfire detection and distance estimation using aerial visible-infrared images10.1109/TIE.2024.3387089 ↗
44. Hierarchical question-image co-attention for visual question answering
45. Berman, M., Triki, A.R., and Blaschko, M.B. (2018, January 18–23). The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00464 ↗
46. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
47. RGB-T semantic segmentation with location, activation, and sharpening10.1109/TCSVT.2022.3208833 ↗
48. Zhang, Q., Zhao, S., Luo, Y., Zhang, D., Huang, N., and Han, J. (2021, January 20–25). ABMDRNet: Adaptive-weighted bi-directional modality difference reduction network for RGB-T semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00266 ↗
49. Context-aware interaction network for RGB-T semantic segmentation10.1109/TMM.2023.3349072 ↗