V-PRUNE: Semantic-Aware Patch Pruning Before Tokenization in Vision–Language Model Inference
Source abstract
Abstract
Recent vision–language models (VLMs) achieve strong performance across multimodal benchmarks but suffer from high inference costs due to the large number of visual tokens. Prior studies have shown that many image tokens receive consistently low attention scores during inference, indicating that a substantial portion of visual content contributes little to final predictions. These observations raise questions about the efficiency of conventional token pruning strategies, which are typically applied after all attention operations and depend on late-emerging attention scores. To address this, we propose V-PRUNE, a semantic-aware patch-level pruning framework for vision–language models that removes redundant content before tokenization. By evaluating local similarity via color and histogram statistics, our method enables lightweight and interpretable pruning without architectural changes. Applied to CLIP-based models, our approach reduces FLOPs and inference time across vision–language understanding tasks, while maintaining or improving accuracy. Qualitative results further confirm that essential regions are preserved and the pruning behavior is human-aligned, making our method a practical solution for efficient VLM inference.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Hanyang University, Seoul 04763, Republic of Korea
Funding
Institute of Information and communications Technology Planning and evaluation (IITP) · National Research Foundation of Korea (NRF)
References (28)
Show source-provided references
1. Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (Trans. Mach. Learn. Res., 2022). CoCa: Contrastive Captioners are Image-Text Foundation Models, Trans. Mach. Learn. Res., preprint.
2. Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. (2024, January 16–22). Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01370 ↗
3. Karthik, S., Roth, K., Mancini, M., and Akata, Z. (2024, January 7–11). Vision-by-Language for Training-Free Compositional Image Retrieval. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
4. Fayyaz, M., Koohpayegani, S.A., Jafari, F.R., Sengupta, S., Joze, H.R.V., Sommerlade, E., Pirsiavash, H., and Gall, J. (2022, January 23–27). Adaptive token sampling for efficient vision transformers. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20083-0_24 ↗
5. Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. (2022, January 23–27). An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.
6. Zhang, Q., Cheng, A., Lu, M., Zhuo, Z., Wang, M., Cao, J., Guo, S., She, Q., and Zhang, S. (2024). [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster. arXiv.
7. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2024, January 7). MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
8. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., and Gao, J. (2020, January 7–12). Unified Vision-Language Pre-Training for Image Captioning and VQA. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.7005 ↗
9. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the International Conference on Learning Representations, Virtual.
10. Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.J. (2021, January 6–14). DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
11. Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. (2022). Token merging: Your vit but faster. arXiv.
12. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Virtual.
13. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Virtual.
14. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
15. Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., and Tao, D. (2022, January 18–24). Patch slimming for efficient vision transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01185 ↗
16. The chi-square test of independence10.11613/BM.2013.018 ↗
17. Liu, H., Li, C., Li, Y., and Lee, Y.J. (2024, January 16–22). Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02484 ↗
18. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y.J. (2024, January 30). LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge. Available online: https://llava-vl.github.io/blog/2024-01-30-llava-next/.
19. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
20. Wang, W., Ren, Y., Luo, H., Li, T., Yan, C., Chen, Z., Wang, W., Li, Q., Lu, L., and Zhu, X. (October, January 29). The all-seeing project v2: Towards general relation comprehension of the open world. Proceedings of the European Conference on Computer Vision, Milan, Italy.
21. Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. (2016, January 11–14). A diagram is worth a dozen images. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part IV 14.10.1007/978-3-319-46493-0_15 ↗
22. Zhang, Z., Wu, H., Li, C., Zhou, Y., Sun, W., Min, X., Chen, Z., Liu, X., Lin, W., and Zhai, G. A-Bench: Are LMMs Masters at Evaluating AI-generated Images? In Proceedings of the Thirteenth International Conference on Learning Representations, Singapore, 24–28 April 2025.
23. Huang, Y., Yuan, Q., Sheng, X., Yang, Z., Wu, H., Chen, P., Yang, Y., Li, L., and Lin, W. (2024). AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception. arXiv.
24. Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., and Yacoob, Y. (2024, January 16–22). Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01363 ↗
25. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.R. (2023, January 6–10). Evaluating Object Hallucination in Large Vision-Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.20 ↗
26. Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. (2022, January 23–27). A-okvqa: A benchmark for visual question answering using world knowledge. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20074-8_9 ↗
27. Zhang, J., Huang, W., Ma, Z., Michel, O., He, D., Gupta, T., Ma, W.C., Farhadi, A., Kembhavi, A., and Krishna, R. (2024, January 10–15). Task Me Anything. Proceedings of the Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024, Vancouver, BC, Canada.
28. Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., and Jia, J. (2025, January 10–17). Visionzip: Longer is better but not necessary in vision language models. Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA.10.1109/CVPR52734.2025.01843 ↗