AdaSeViLA: Adaptive Dynamic Temporal Window and Object-Aware Frame Selection for Video Question Answering
Source abstract
Abstract
Video question answering remains a challenging task that requires a sophisticated understanding of both visual content and temporal dynamics across video sequences. Current approaches typically rely on fixed temporal processing strategies and uniform frame-selection mechanisms, which fail to adapt to the diverse requirements of different question types and may overlook critical visual information. We propose AdaSeViLA, an adaptive framework that enhances video understanding through two key innovations: Adaptive Temporal Window Selection (ATWS) that dynamically adjusts the number of processed frames (3–12 frames) based on question-type classification, and Object-importance-Aware Frame Selection (OAFS) that combines global relevance with local visual saliency for enhanced frame identification. Our approach intelligently allocates computational resources based on question complexity while maintaining high accuracy through improved frame-selection mechanisms. Extensive experiments on three challenging VideoQA benchmarks demonstrate that AdaSeViLA achieves superior performance: 87.4% accuracy on MM-AU (+2.7% over SeViLA), 73.6% on NExT-QA (+0.4% improvement), and 61.6% on STAR (+0.6% gain), while providing up to 4× computational speedup for short-term tasks. These results validate the effectiveness of adaptive temporal processing and object-aware selection in advancing video question answering capabilities.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Beihang University, Beijing 100191, China
Autonomous Driving Department, Xiaomi Corporation, Beijing 102600, China
Hangzhou International Innovation Institute, Beihang University, Beijing 100191, China
School of Software Engineering, Beihang University, Beijing 100191, China
Funding
National Key R&D Program of China · Research Start-up Funds of Hangzhou International Innovation Institute of Beihang University
References (29)
Show source-provided references
1. Xiao, J., Shang, X., Yao, A., and Chua, T.S. (2021, January 19–25). Next-qa: Next phase of question-answering to explaining temporal actions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00965 ↗
2. Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. (2021, January 11–17). Just ask: Learning to answer questions from millions of narrated videos. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00171 ↗
3. Gao, D., Zhou, L., Ji, L., Zhu, L., Yang, Y., and Shou, M.Z. (2023, January 17–24). Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01419 ↗
4. Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of the International Conference on Machine Learning, JMLR.org.
5. Self-chained image-language model for video localization and question answering
6. Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., and Liu, J. (2021, January 19–25). Less is more: Clipbert for video-and-language learning via sparse sampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00725 ↗
7. Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., and Niebles, J.C. (2022, January 18–24). Revisiting the “video” in video-language understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00293 ↗
8. Momeni, L., Caron, M., Nagrani, A., Zisserman, A., and Schmid, C. (2023, January 1–6). Verbs in action: Improving verb understanding in video-language models. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01428 ↗
9. Kim, S., Kim, J.H., Lee, J., and Seo, M. (2023). Semi-parametric video-grounded text generation. arXiv.
10. Xiao, J., Zhou, P., Chua, T.S., and Yan, S. (2022). Video graph transformer for video question answering. Proceedings of the European Conference on Computer Vision, Springer.10.1007/978-3-031-20059-5_3 ↗
11. Ye, Q., Xu, G., Yan, M., Xu, H., Qian, Q., Zhang, J., and Huang, F. (2023, January 1–6). Hitea: Hierarchical temporal-aware video-language pre-training. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01413 ↗
12. Wu, B., Yu, S., Chen, Z., Tenenbaum, J.B., and Gan, C. (2024). Star: A benchmark for situated reasoning in real-world videos. arXiv.
13. Fang, J., Li, L.l., Zhou, J., Xiao, J., Yu, H., Lv, C., Xue, J., and Chua, T.S. (2024, January 16–22). Abductive Ego-View Accident Video Understanding for Safe Driving Perception. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02080 ↗
14. Le, T.M., Le, V., Venkatesh, S., and Tran, T. (2020, January 13–19). Hierarchical conditional relation networks for video question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00999 ↗
15. Min, J., Buch, S., Nagrani, A., Cho, M., and Schmid, C. (2024). MoReVQA: Exploring modular reasoning models for video question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF.10.1109/CVPR52733.2024.01257 ↗
16. VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset10.1109/TPAMI.2024.3479776 ↗
17. Rizve, M.N., Fei, F., Unnikrishnan, J., Tran, S., Yao, B., Zeng, B., Shah, M., and Chilimbi, T.M. (2024, January 16–22). VidLA: Video-Language Alignment at Scale. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01332 ↗
18. Li, S., Li, L., Liu, Y., Ren, S., Liu, Y., Gao, R., Sun, X., and Hou, L. (2024). Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. Proceedings of the European Conference on Computer Vision, Springer.10.1007/978-3-031-72897-6_19 ↗
19. Flamingo: A visual language model for few-shot learning10.52202/068431-1723 ↗
20. Wang, X., Liang, J., Wang, C.K., Deng, K., Lou, Y., Lin, M.C., and Yang, S. (2023). VLAP: Efficient Video-Language Alignment via Frame Prompting and Distilling for Video Question Answering. arXiv.10.1007/978-3-031-73033-7_11 ↗
21. Yang, X., Li, Z., Xu, H., Zhang, H., Ye, Q., Li, C., Yan, M., Zhang, Y., Huang, F., and Huang, S. (2023, January 1–6). Learning Trajectory-Word Alignments for Video-Language Tasks. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00237 ↗
22. Contrastive video question answering via video graph transformer10.1109/TPAMI.2023.3292266 ↗
23. Bert: Pre-training of deep bidirectional transformers for language understanding
24. He, P., Liu, X., Gao, J., and Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv.
25. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
26. Scaling instruction-finetuned language models
27. Xiao, J., Yao, A., Li, Y., and Chua, T.S. (2024, January 16–22). Can I Trust Your Answer? Visually Grounded Video Question Answering. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01254 ↗
28. Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K.Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., and Shan, Y. (2023, January 17–24). All in one: Exploring unified video-language pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00638 ↗
29. Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., and Wang, Z. (2022). Internvideo: General video foundation models via generative and discriminative learning. arXiv.