Feasibility of Visual Question Answering (VQA) for Post-Disaster Damage Detection Using Aerial Footage
Source abstract
Abstract
Natural disasters are a major source of significant damage and costly repairs around the world. After a natural disaster occurs, there is usually a significant amount of damage, and with that, there are also a lot of costs involved with repairing and aiding all the people involved. In addition, the occurrence of natural phenomena has increased significantly in the past decade. With that in mind, post-disaster damage detection is usually performed manually by human operators. Taking into consideration all the areas one has to closely look into, as well as the difficult terrain and places with hard access, it becomes easy to understand how incredibly difficult it is for a surveyor to identify and annotate every single possible damage out there. Because of that, it has become essential to find new creative solutions for damage detection and classification in the case of natural disasters, especially hurricanes. This study focuses on the feasibility of using a Visual Question Answering (VQA) method for post-disaster damage detection, using aerial footage taken from an Unmanned Aerial Vehicle (UAV). Two other approaches are also utilized to provide comparison and to evaluate the performance of VQA. Our case study on our custom dataset collected after Hurricane Sally shows successful results using VQA for post-disaster damage detection applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Electrical and Computer Engineering, University of West Florida, Pensacola, FL 32514, USA
Intelligent Systems & Robotics, University of West Florida, Pensacola, FL 32514, USA
References (38)
Show source-provided references
1. Munich, R.E. (2021, March 16). Record Hurricane Season and Major Wildfires—The Natural Disaster Figures for 2020. Available online: https://www.munichre.com/en/company/media-relations/media-information-and-corporate-news/media-information/2021/2020-natural-disasters-balance.html.
2. Dapena, K. (2021, March 16). The Rising Costs of Hurricanes. Available online: https://www.wsj.com/articles/the-rising-costs-of-hurricanes-1538222400.
3. Example-based object detection in images by components10.1109/34.917571 ↗
4. Omni Directional Moving Object Detection and Tracking with Virtual Reality Feedback
5. Yin, T., Zhou, X., and Krahenbuhl, P. (2021, January 20–25). Center-Based 3D Object Detection and Tracking. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01161 ↗
6. Viola, P., and Jones, M. (2001, January 8–14). Rapid Object Detection Using a Boosted Cascade of Simple Features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA.
7. Lienhart, R., Kuranov, A., and Pisarevsky, V. (2003). Joint Pattern Recognition Symposium, Springer.
8. Wang, X., Yang, M., Zhu, S., and Lin, Y. (2013, January 1–8). Regionlets for Generic Object Detection. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.10 ↗
9. Zhu, X., Liang, J., and Hauptmann, A. (2021, January 3–8). Msnet: A Multilevel Instance Segmentation Network for Natural Disaster Damage Assessment in Aerial Videos. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00207 ↗
10. Nex, F., Duarte, D., Steenbeek, A., and Kerle, N. (2019). Towards real-time building damage mapping with low-cost UAV solutions. Remote Sens., 11.10.3390/rs11030287 ↗
11. Pi, Y., Nath, N., and Behzadan, A.H. (2020). Disaster impact information retrieval using deep learning object detection in crowdsourced drone footage. Proc. Int. Workshop Intell. Comput. Eng., 134–143.
12. Cai, Z., and Vasconcelos, N. (2018, January 18–23). Cascade R-CNN: Delving into High Quality Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00644 ↗
13. Zhou, B., Tian, Y., Sukhbaatar, S., Szlam, A., and Fergus, R. (2015). Simple Baseline for Visual Question Answering. arXiv.
14. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, L., and Parikh, D. (2015, January 7–13). VQA: Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.279 ↗
15. Chen, K., Wang, J., Chen, L.-C., Gao, H., Xu, W., and Nevatia, R. (2015). Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv.
16. Shih, K.J., Singh, S., and Hoiem, D. (2016, January 27–30). Where to Look: Focus Regions for Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.499 ↗
17. Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. (2016, January 27–30). Stacked Attention Networks for Image Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.10 ↗
18. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016, January 1–5). Multimodal compact bilinear pooling for visual question answering and visual grounding. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TA, USA.10.18653/v1/D16-1044 ↗
19. Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., and Zhang, B.-T. (2016). Hadamard Product for Low-Rank Bilinear Pooling. arXiv.
20. Yu, Z., Yu, J., Fan, J., and Tao, D. (2017, January 22–29). Multi-Modal Factorized Bilinear Pooling with Coattention Learning for Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.202 ↗
21. Lu, J., Yang, J., Batra, D., and Parikh, D. (2016, January 5–10). Hierarchical Question-Image Co-Attention for Visual Question Answering. Proceedings of the NeurIPS, Barcelona, Spain.
22. RSVQA: Visual question answering for remote sensing data10.1109/TGRS.2020.2988782 ↗
23. A novel feature extractor for human action recognition in visual question answering10.1016/j.patrec.2021.04.002 ↗
24. Vo, H.Q., Phung, T.H., and Ly, N.Q. (2020, January 26–27). VQASTO: Visual Question Answering System for Action Surveillance Based on Task Ontology. Proceedings of the 2020 7th NAFOSTED Conference on Information and Computer Science (NICS), Ho Chi Minh City, Vietnam.10.1109/NICS51282.2020.9335891 ↗
25. Abacha, A.B., Hasan, S.A., Datla, V.V., Liu, J., Demner-Fushman, D., and Müller, H. (2019, January 9–12). VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF. Proceedings of the CLEF 2019—Conference and Labs of the Evaluation Forum, Lugano, Switzerland.
26. Hela, V., Lubna, A., and Kalady, S. (2022, January 24–26). CarParkingVQA: Visual Question Answering Application on Car Parking Occupancy Detection. Proceedings of the 2022 IEEE 19th India Council International Conference (INDICON), Kochi, India.10.1109/INDICON56171.2022.10039925 ↗
27. A survey of convolutional neural networks: Analysis, applications, and prospects10.1109/TNNLS.2021.3084827 ↗
28. Sarkar, A., and Rahnemoonfar, M. (2021). VQA-Aid: Visual Question Answering for Post-Disaster Damage Assessment and Analysis. arXiv.10.1109/IGARSS47720.2021.9553578 ↗
29. Sasha, S. (2023, April 10). A Comprehensive Guide to Convolutional Neural Networks—The eli5 Way. Medium. Towards Data Science. Available online: https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53.
30. Gad, A.F. (2023, April 10). Faster R-CNN Explained for Object Detection Tasks. Paperspace Blog. Available online: https://blog.paperspace.com/faster-r-cnn-explained-object-detection/.
31. Chowdhury, T., Rahnemoonfar, M., Murphy, R., and Fernandes, O. (2020, January 10-13). Comprehensive Semantic Segmentation on High Resolution UAV Imagery for Natural Disaster Damage Assessment. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA.10.1109/BigData50022.2020.9377916 ↗
32. Agrawal, A., Kembhavi, A., Batra, D., and Parikh, D. (2017). C-vqa: A Compositional Split of the Visual Question Answering (VQA) v1. 0 Dataset. arXiv.10.1007/s11263-016-0966-6 ↗
33. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017, January 21–26). Making the v in vqa Matter: Elevating the Role of Image Understanding in Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.670 ↗
34. Kant, Y., Moudgil, A., Batra, D., Parikh, D., and Agrawal, H. (2021, January 10–17). Contrast and Classify: Training Robust VQA Models. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00163 ↗
35. Visualqa (2022, January 17). VQA: Visual Question Answering. Available online: https://visualqa.org/download.html.
36. Jason, B. (2022, January 10). A Gentle Introduction to the Bag-of-Words Model. Machine Learning Mastery. Available online: https://machinelearningmastery.com/gentle-introduction-bag-words-model/.
37. Zhou, V. (2022, January 10). A Simple Explanation of the Bag-of-Words Model. Available online: https://victorzhou.com/blog/bag-of-words/.
38. Lowande, R. (2022). Visual Question Answering (VQA) Analyses for Post-Disaster Damage Detection and Identification Using Aerial Footage. [Master’s Thesis, University of West Florida].