PoMQ-ViT: Mixed-Precision Quantization Vision Transformer with Pareto Optimization
Source abstract
Abstract
Vision Transformer (ViT) has established itself as the leading model in image processing, and quantization offers a practical way to deploy ViTs on memory-constrained devices. This technique maps float32 inference to lower bits such as int8. However, existing studies often use uniform bit-width across ViT encoders, leading to sub-optimal allocation and accuracy loss. This study aims to optimize bit-width allocation by leveraging the observed variation in quantization sensitivity among different ViT encoders. We propose PoMQ-ViT, a method that dynamically assigns bit-width to ViT encoders based on their sensitivity. It uses Pareto Optimization to balance accuracy and inference speed. Experiments on the ImageNet Dataset show PoMQ-ViT outperforms uniform bit-width methods across DeiT and Swin models. Within the same computation budget, it also achieves 0.1–1.5 percentage points higher accuracy than other mixed-precision paradigms across different models. This work demonstrates PoMQ-ViT’s effectiveness in ViT quantization, providing a practical solution for resource-constrained deployments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Digital Steel, Northeastern University, Shenyang 110819, China
References (41)
Show source-provided references
1. Attention is all you need
2. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. Adv. Neural Inf. Process. Syst., 34.
3. Training data-efficient image transformers distillation through attention
4. End-to-end object detection with transformers
5. Transformers in small object detection: A benchmark and survey of state-of-the-art
6. Beal, J., Kim, E., Tzeng, E., Park, D.H., Zhai, A., and Kislyuk, D. (2020). Toward transformer-based object detection. arXiv.
7. A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images10.1007/s11517-019-02008-8 ↗
8. Yigit, G. (2023). Vit-GAN: Image-to-Image Translation with Vision Transformes and Conditional GANS, Authorea Preprints, Springer. Available online: https://arxiv.org/abs/2110.09305.
9. Transformer-based conditional generative adversarial networks for image generation
10. Tang, Y., Wang, Y., Guo, J., Tu, Z., Han, K., Hu, H., and Tao, D. (2024). A survey on transformer compression. arXiv.
11. Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., and Shen, C. (2023). A survey on efficient training of transformers. arXiv.10.24963/ijcai.2023/764 ↗
12. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (2016). DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv.
13. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018, January 18–23). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00286 ↗
14. Quantized neural networks: Training neural networks with low precision weights and activations
15. Not all patches are what you need: Expediting vision transformers via token reorganizations
16. Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., and Sun, X. (March, January 22). Evo-vit: Slow-fast token evolution for dynamic vision transformer. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual. Available online: https://arxiv.org/pdf/2108.01390.
17. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). Tinybert: Distilling bert for natural language understanding. arXiv.10.18653/v1/2020.findings-emnlp.372 ↗
18. Wu, K.T., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., and Tinyvit, L.Y. (2022). Fast Pretraining Distillation for Small Vision Transformers. Lect. Notes Comput. Sci., 13681. Available online: https://arxiv.org/pdf/2207.10666v1.
19. Yang, Z., Li, Z., Zeng, A., Li, Z., Yuan, C., and Li, Y. (2022). Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv.
20. Yang, H., Yin, H., Shen, M., Molchanov, P., Li, H., and Kautz, J. (2023, January 17–24). Global vision transformer pruning with hessian-aware saliency. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01779 ↗
21. Lin, Y., Zhang, T., Sun, P., Li, Z., and Zhou, S. (2021). Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv.10.24963/ijcai.2022/164 ↗
22. Li, Z., and Gu, Q. (2023, January 2–3). I-vit: Integer-only quantization for efficient vision transformer inference. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01565 ↗
23. Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., and Modha, D.S. (2019). Learned step size quantization. arXiv.
24. Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., and Gu, S. (2021). Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv.
25. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 10–17). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
26. Pvt v2: Improved baselines with pyramid vision transformer10.1007/s41095-022-0274-8 ↗
27. Kitaev, N., Kaiser, Ł., and Levskaya, A. (2020). Reformer: The efficient transformer. arXiv.
28. Fast vision transformers with hilo attention
29. Mehta, S., and Rastegari, M. (2022). Separable self-attention for mobile vision transformers. arXiv.
30. Towards lightweight transformer via group-wise transformation for vision-and-language tasks10.1109/TIP.2021.3139234 ↗
31. Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S.W., Anwer, R.M., and Shahbaz Khan, F. (2022). Edgenext: Efficiently amalgamated cnn-transformer architecture for mobile vision applications. Eur. Conf. Comput. Vis., 3–20.10.1007/978-3-031-25082-8_1 ↗
32. Yang, C., Wang, Y., Zhang, J., Zhang, H., Wei, Z., Lin, Z., and Yuille, A. (2022, January 18–24). Lite vision transformer with enhanced self-attention. Proceedings of the IEEE/CVF International Conference on Computer Vision, New Orleans, LA, USA.10.1109/CVPR52688.2022.01169 ↗
33. He, Y., Lou, Z., Zhang, L., Liu, J., Wu, W., Zhou, H., and Zhuang, B. (2023, January 2–6). Bivit: Extremely compressed binary vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France. Available online: https://arxiv.org/pdf/2211.07091.
34. Li, Z., Xiao, J., Yang, L., and Gu, Q. (2023, January 2–6). Repq-vit: Scale reparameterization for post-training quantization of vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01580 ↗
35. Liu, Y., Yang, H., Dong, Z., Keutzer, K., Du, L., and Zhang, S. (2023, January 2–6). Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France. Available online: https://arxiv.org/abs/2211.16056.
36. Nagel, M., Amjad, R.A., Van Baalen, M., Louizos, C., and Blankevoort, T. (2020, January 13–18). Up or down? adaptive rounding for post-training quantization. Proceedings of the International Conference on Machine Learning, Virtual. Available online: https://proceedings.mlr.press/v119/nagel20a.html.
37. Choukroun, Y., Kravchik, E., Yang, F., and Kisilev, P. (2019, January 27–28). Low-bit quantization of neural networks for efficient inference. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW.2019.00363 ↗
38. Ngatchou, P., Zarei, A., and El-Sharkawi, A. (2005, January 6–10). Pareto multi objective optimization. Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems, Arlington, VA, USA.
39. Bechikh, S., Datta, R., and Gupta, A. (2017). Recent Advances in Evolutionary Multi-Objective Optimization, Springer.10.1007/978-3-319-42978-6 ↗
40. Liashchynskyi, P., and Liashchynskyi, P. (2019). Grid search, random search, genetic algorithm: A big comp arison for NAS. arXiv.
41. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada. Available online: https://arxiv.org/abs/2103.14030.