Cross-Modal Adaptive Fusion and Multi-Scale Aggregation Network for RGB-T Crowd Density Estimation and Counting
Source abstract
Abstract
Crowd counting is a significant task in computer vision. By combining the rich texture information from RGB images with the insensitivity to illumination changes offered by thermal imaging, the applicability of models in real-world complex scenarios can be enhanced. Current research on RGB-T crowd counting primarily focuses on feature fusion strategies, multi-scale structures, and the exploration of novel network architectures such as Vision Transformer and Mamba. However, existing approaches face two key challenges: limited robustness to illumination shifts and insufficient handling of scale discrepancies. To address these challenges, this study aims to develop a robust RGB-T crowd counting framework that remains stable under illumination shifts, through introduces two key innovations beyond existing fusion and multi-scale approaches: (1) a cross-modal adaptive fusion module (CMAFM) that actively evaluates and fuses reliable cross-modal features under varying scenarios by simulating a dynamic feature selection and trust allocation mechanism; and (2) a multi-scale aggregation module (MSAM) that unifies features with different receptive fields to an intermediate scale and performs weighted fusion to enhance modeling capability for cross-modal scale variations. The proposed method achieves relative improvements of 1.57% in GAME(0) and 0.78% in RMSE on the DroneRGBT dataset compared to existing methods, and improvements of 2.48% and 1.59% on the RGBT-CC dataset, respectively. It also demonstrates higher stability and robustness under varying lighting conditions. This research provides an effective solution for building stable and reliable all-weather crowd counting systems, with significant application prospects in smart city security and management.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Fire Detection and Safety Systems · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Robot Brain Technology Innovation Center, Tianfu Jiangxi Laboratory, Chengdu 610041, China
College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China
Funding
Sichuan Science and Technology Program
References (50)
Show source-provided references
1. Khan, A., Ali Shah, J., Kadir, K., Albattah, W., and Khan, F. (2020). Crowd Monitoring and Localization Using Deep Convolutional Neural Network: A Review. Appl. Sci., 10.10.3390/app10144781 ↗
2. Crowd management: The overlooked component of smart transportation systems10.1109/MCOM.2019.1800641 ↗
3. Locate, size, and count: Accurately resolving people in dense crowds via detection
4. Khan, M.A., Menouar, H., and Hamila, R. (2023). Revisiting crowd counting: State-of-the-art, trends, and future perspectives. Image Vis. Comput., 129.10.1016/j.imavis.2022.104597 ↗
5. Alhawsawi, A.N., Khan, S.D., and Ur Rehman, F. (2024). Crowd Counting in Diverse Environments Using a Deep Routing Mechanism Informed by Crowd Density Levels. Information, 15.10.3390/info15050275 ↗
6. Luo, H., Sang, J., Wu, W., Xiang, H., Xiang, Z., Zhang, Q., and Wu, Z. (2018). A High-Density Crowd Counting Method Based on Convolutional Feature Fusion. Appl. Sci., 8.10.3390/app8122367 ↗
7. Ilyas, N., Lee, B., and Kim, K. (2021). HADF-Crowd: A Hierarchical Attention-Based Dense Feature Extraction Network for Single-Image Crowd Counting. Sensors, 21.10.3390/s21103483 ↗
8. A survey of recent advances in cnn-based single image crowd counting and density estimation10.1016/j.patrec.2017.07.007 ↗
9. Liu, Z., and Yaermaimaiti, Y. (2025). A study of pedestrian detection algorithms for use in complex environments. Eng. Res. Express, 7.10.1088/2631-8695/adfc28 ↗
10. Neural Network-Based Crowd Counting Systems: State of the Art, Challenges, and Perspectives
11. Peng, T., Li, Q., and Zhu, P. (2020–4, January 30). RGB-T Crowd Counting from Drone: A Benchmark and MMCCN Network. Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan.10.1007/978-3-030-69544-6_30 ↗
12. Liu, L., Chen, J., Wu, H., Li, G., Li, C., and Lin, L. (2021, January 20–25). Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00479 ↗
13. Gu, S., and Lian, Z. (2023). A unified RGB-T crowd counting learning framework. Image Vis. Comput., 131.10.1016/j.imavis.2023.104631 ↗
14. DEFNet: Dual-branch enhanced feature fusion network for RGB-T crowd counting10.1109/TITS.2022.3203385 ↗
15. MC3Net: Multimodality Cross-Guided Compensation Coordination Network for RGB-T Crowd Counting10.1109/TITS.2023.3321328 ↗
16. A three-stream fusion and self-differential attention network for multi-modal crowd counting10.1016/j.patrec.2024.04.025 ↗
17. MIANet: Bridging the Gap in Crowd Density Estimation with Thermal and RGB Interaction10.1109/TITS.2024.3478292 ↗
18. CCANet: A Collaborative Cross-Modal Attention Network for RGB-D Crowd Counting10.1109/TMM.2023.3262978 ↗
19. Kong, W., Liu, J., Hong, Y., Li, H., and Shen, J. (2024). Cross-modal collaborative feature representation via Transformer-based multi-modal mixers for RGB-T crowd. Expert Syst. Appl., 255.10.1016/j.eswa.2024.124483 ↗
20. Kong, W., Yu, Z., Li, H., and Zhang, J. (2024). Cross-modal misalignment-robust feature fusion for crowd counting. Eng. Appl. Artif. Intell., 136.10.1016/j.engappai.2024.108898 ↗
21. MISF-Net: Modality-Invariant and -Specific Fusion Network for RGB-T Crowd Counting10.1109/TMM.2025.3535330 ↗
22. Redesigning multi-scale neural network for crowd counting10.1109/TIP.2023.3289290 ↗
23. Gong, S., Yao, Z., Zuo, W., Yang, J., Yuen, P., and Zhang, S. (2025). Spatially adaptive pyramid feature fusion for scale-aware crowd counting. Pattern Recognit., 168.10.1016/j.patcog.2025.111832 ↗
24. Wang, T., Zhang, T., Zhang, K., Wang, H., Li, M., and Lu, J. (2023). Context attention fusion network for crowd counting. Knowl. Based Syst., 271.10.1016/j.knosys.2023.110541 ↗
25. Zhu, A., Duan, G., Zhu, X., Zhao, L., Huang, Y., Hua, G., and Snoussi, H. (2021). CDADNet: Context-guided dense attentional dilated network for crowd counting. Signal Process.-Image Commun., 98.10.1016/j.image.2021.116379 ↗
26. CCST: Crowd counting with swin transformer10.1007/s00371-022-02485-3 ↗
27. VMamba-Crowd: Bridging multi-scale features from Visual Mamba for weakly-supervised crowd counting10.1016/j.patrec.2025.08.005 ↗
28. Kernel-based density map generation for dense object counting10.1109/TPAMI.2020.3022878 ↗
29. Counting varying density crowds through density guided adaptive selection CNN and transformer estimation10.1109/TCSVT.2022.3208714 ↗
30. Focal inverse distance transform maps for crowd localization10.1109/TMM.2022.3203870 ↗
31. Ma, Z., Wei, X., Hong, X., and Gong, Y. (November, January 27). Bayesian loss for crowd count estimation with point supervision. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
32. Song, Q., Wang, C., Jiang, Z., Wang, Y., Tai, Y., Wang, C., Li, J., Huang, F., and Wu, Y. (2021, January 10–17). Rethinking counting and localization in crowds: A purely point-based framework. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00335 ↗
33. Modeling Noisy Annotations for Point-Wise Supervision10.1109/TPAMI.2023.3299753 ↗
34. Li, H., Zhang, J., Kong, W., Shen, J., and Shao, Y. (2023). CSA-Net: Cross-modal scale-aware attention-aggregated network for RGB-T crowd counting. Expert Syst. Appl., 213.10.1016/j.eswa.2022.119038 ↗
35. Consistency-constrained RGB-T crowd counting via mutual information maximization10.1007/s40747-024-01427-x ↗
36. Pan, Y., Zhou, W., Fang, M., and Qiang, F. (2024). Graph enhancement and transformer aggregation network for RGB-thermal crowd counting. IEEE Geosci. Remote Sens. Lett., 21.10.1109/LGRS.2024.3362820 ↗
37. Chen, Y., Zhao, H., Huang, L., Yang, Y., Kang, W., and Zhang, J. (2025). TransMambaCC: Integrating Transformer and Pyramid Mamba Network for RGB-T Crowd Counting. Appl. Intell., 55.10.1007/s10489-025-06912-5 ↗
38. Kong, W., Yu, Z., Li, H., Tong, L., Zhao, F., and Li, Y. (2024). CrowdAlign: Shared-weight dual-level alignment fusion for RGB-T crowd counting. Image Vis. Comput., 148.10.1016/j.imavis.2024.105152 ↗
39. Kuang, X., and Tao, B. (2024). ODGNet: Robotic Grasp Detection Network Based on Omni-Dimensional Dynamic Convolution. Appl. Sci., 14.10.3390/app14114653 ↗
40. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). CBAM: Convolutional Block Attention Module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
41. Towards Understanding Convergence and Generalization of AdamW10.1109/TPAMI.2024.3382294 ↗
42. The WuC-Adam algorithm based on joint improvement of Warmup and cosine annealing algorithms10.3934/mbe.2024054 ↗
43. Pan, Y., Zhou, W., Qian, X., Mao, S., Yang, R., and Yu, L. (2023). CGINet: Cross-modality grade interaction network for RGB-T crowd counting. Eng. Appl. Artif. Intell., 126.10.1016/j.engappai.2023.106885 ↗
44. Xie, Z., Shao, F., Mu, B., Chen, H., Jiang, Q., Lu, C., and Ho, Y.S. (2024). BGDFNet: Bidirectional Gated and Dynamic Fusion Network for RGB-T Crowd Counting in Smart City System. IEEE Trans. Instrum. Meas., 73.10.1109/TIM.2024.3418111 ↗
45. Zhang, Y., Choi, S., and Hong, S. (2025). Memory-efficient cross-modal attention for RGB-X segmentation and crowd counting. Pattern Recognit., 162.10.1016/j.patcog.2025.111376 ↗
46. Modal-Adaptive Spatial-Aware-Fusion and Propagation Network for Multimodal Vision Crowd Counting10.1109/TCE.2025.3571571 ↗
47. Fine-Grained Crowd Counting10.1109/TIP.2021.3049938 ↗
48. Category-Contrastive Fine-Grained Crowd Counting and Beyond10.1109/TMM.2024.3521823 ↗
49. Superpixel-Guided Graph-Attention Boundary GAN for Adaptive Feature Refinement in Scribble-Supervised Medical Image Segmentation
50. Extended Feature Pyramid Network for Small Object Detection10.1109/TMM.2021.3074273 ↗