Multimodal Natural Disaster Scene Recognition with Integrated Large Model and Mamba
Source abstract
Abstract
The accurate identification of natural disasters is crucial in ensuring effective post-disaster relief efforts. However, the existing models for disaster classification often incur high costs. To address this, we propose leveraging the most advanced pre-trained large language models, which offer superior generative and multimodal understanding capabilities. Using a question-answering approach, we extract textual descriptions and category prediction probabilities for disaster scenarios, which are then used as input to our proposed Mamba Multimodal Disaster Recognition Network (Mamba-MDRNet). This model integrates a large pre-trained model with the Mamba mechanism, enabling the selection of the most reliable modality information as a robust basis for scene classification. Extensive experiments demonstrate consistent performance improvements across various visual models with heterogeneous architectures. Notably, integrating EfficientNet within Mamba-MDRNet yielded 97.82% accuracy for natural scene classification, surpassing the performance of the CNN (91.75%), ViT (94.50%), and ResNet18 (97.25%). These results highlight the potential of multimodal models combining large models and the Mamba mechanism for disaster type prediction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote Sensing and Land Use
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Science, North China University of Technology, Beijing 100144, China
Funding
Yuxiu Innovation Project of NCUT
References (39)
Show source-provided references
1. Persistent effects of natural disasters on human development: Quasi-experimental evidence for Argentina10.1007/s10668-020-01064-7 ↗
2. Weber, E., Marzo, N., Papadopoulos, D.P., Biswas, A., Lapedriza, A., Ofli, F., Imran, M., and Torralba, A. (2020). Detecting natural disasters, damage, and incidents in the wild. Computer Vision–ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23–28 August 2020, Springer. Proceedings, Part XIX 16.10.1007/978-3-030-58529-7_20 ↗
3. Deep learning in remote sensing applications: A meta-analysis and review10.1016/j.isprsjprs.2019.04.015 ↗
4. Unsupervised change detection analysis in satellite image time series using deep learning combined with graph-based approaches10.1109/JSTARS.2020.2982631 ↗
5. A survey on the applications of convolutional neural networks for synthetic aperture radar: Recent advances10.1109/MAES.2021.3117369 ↗
6. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A survey of large language models. arXiv.
7. Yang, H., Zhao, Y., Wu, Y., Wang, S., Zheng, T., Zhang, H., Che, W., and Qin, B. (2024). Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey. arXiv.
8. Vision-Enabled Large Language and Deep Learning Models for Image-Based Emotion Recognition10.1007/s12559-024-10281-5 ↗
9. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv.
10. Yang, W., Li, C., Zhang, J., and Zong, C. (2023). Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv.
11. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-B., Soricut, R., Lazaridou, A., Firat, O., and Schrittwieser, J. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv.
12. Texture Classification Using ResNet and EfficientNet
13. Ding, N., Tang, Y., Fu, Z., Xu, C., Han, K., and Wang, Y. (2023). GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks?. arXiv.
14. Johnson, R., and Zhang, T. (2014). Effective Use of Word Order for Text Categorization with Convolutional Neural Networks. arXiv.10.3115/v1/N15-1011 ↗
15. Reducing the dimensionality of data with neural networks10.1126/science.1127647 ↗
16. Understanding bag-of-words model: A statistical framework10.1007/s13042-010-0001-0 ↗
17. Research paper classification systems based on TF-IDF and LDA schemes10.1186/s13673-019-0192-7 ↗
18. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
19. Facial expression recognition and histograms of oriented gradients: A comprehensive study10.1186/s40064-015-1427-3 ↗
20. Ghojogh, B., and Ghodsi, A. (2023). Recurrent neural networks and long short-term memory networks: Tutorial and survey. arXiv.
21. Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems, MIT Press.
22. Vision-language models for vision tasks: A survey10.1109/TPAMI.2024.3369699 ↗
23. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 Technical Report. arXiv.
24. Exploring the limits of transfer learning with a unified text-to-text transformer
25. Zbinden, R. (2022). Implementing and experimenting with diffusion models for text-to-image generation. arXiv.
26. Li, J., Li, D., Xiong, C., and Hoi, S. (2022, January 17–23). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA.
27. Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., and Li, C. (2021). Florence: A new foundation model for computer vision. arXiv.
28. Emotion recognition based on brain-like multimodal hierarchical perception10.1007/s11042-023-17347-w ↗
29. Deep multimodal learning: A survey on recent advances and trends10.1109/MSP.2017.2738401 ↗
30. Wang, X., Wang, S., Ding, Y., Li, Y., Wu, W., Rong, Y., Kong, W., Huang, J., Li, S., and Yang, H. (2024). State space model for new-generation network alternative to transformers: A survey. arXiv.
31. Gu, A., and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv.
32. Jelassi, S., Brandfonbrener, D., Kakade, S.M., and Malach, E. (2024). Repeat after me: Transformers are better than state space models at copying. arXiv.
33. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual.
34. Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. (2020, January 23–28). Uniter: Universal image-text representation learning. Proceedings of the Computer Vision—ECCV 2020, 16th European Conference, Glasgow, UK.10.1007/978-3-030-58577-8_7 ↗
35. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
36. Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
37. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
38. Tan, M. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv.
39. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22–29). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.74 ↗