CGA-ViT: Channel-Guided Additive Attention for Efficient Vision Recognition
Source abstract
Abstract
Vision transformers (ViTs) excel at global context modeling with self-attention. However, standard self-attention leads to quadratic computational complexity, which restricts its practical use in high-resolution or latency-sensitive tasks. Existing methods achieve linear complexity via local window constraints or additive approximations. However, they often compromise long-range dependency modeling. To address this issue, we propose the channel-guided additive attention vision transformer (CGA-ViT), which achieves synergistic optimization of multi-scale feature extraction and efficient global context modeling. First, we propose multi-scale dilated feature embedding (MDFE). By designing multi-scale sampling and spatial feature embedding, we can expand the receptive field and capture fine-grained features simply by adjusting the dilation rate in the early stages; second, we design channel-guided additive attention (CGA), dynamically modulating key vectors using query-derived descriptors, enabling long-range semantic interactions while maintaining linear complexity growth. We adopt a hierarchical structure, and in the shallow layers, we use CGA to carry out local-global interactions and use efficient additive attention in deep layers for global integration. Evaluations on ImageNet-1K show that CGA-ViT achieves 84.0% Top-1 accuracy with 4.7 GFLOPs, outperforming Swin-T (81.3%) and ConvNeXt-T (82.1%) by 2.7 and 1.9 percentage points under comparable computational costs. Ablation experiments verify MDFE and CGA, which together contribute to 65.0% of performance gains, with the rest from token-level supervision. Overall, CGA-ViT effectively balances the intrinsic tradeoff between efficiency and global modeling capability, significantly boosts visual recognition performance without extra computational overhead, and provides an efficient solution for lightweight ViT design.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Memory and Neural Computing · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Electrical Engineering, Hebei University of Engineering, No. 19 Taiji Road, Economic and Technological Development Zone, Handan 056038, China
Information Research Center of Military Science, Academy of Military Sciences, Beijing 100142, China
Mathematics and Physics School, University of Science and Technology Beijing, Beijing 100083, China
References (49)
Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
2. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
3. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A ConvNet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
4. Consistent Discrepancy Learning for Intra-Camera Supervised Person Re-Identification10.1109/TMM.2022.3146775 ↗
5. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
6. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
7. COMO: Efficient Deep Neural Networks Expansion with Convolutional Maxout10.1109/TMM.2020.3002614 ↗
8. He, K., Gkioxari, G., Dollár, P., and Girshick, R.B. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
9. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
10. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016, January 11–14). SSD: Single Shot MultiBox Detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
11. Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv.
12. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
13. Chen, L.-C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv.
14. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully Convolutional Networks for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
15. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
16. Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.-H., and Khan, F.S. (2023, January 1–6). SwiftFormer: Efficient Additive Attention for Transformer-Based Real-Time Mobile Vision Applications. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01598 ↗
17. Efficientformer: Vision transformers at mobilenet speed
18. Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S.W., Anwer, R.M., and Shahbaz Khan, F. (2022, January 23–27). EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-25082-8_1 ↗
19. Mehta, S., and Rastegari, M. (2022). Separable Self-Attention for Mobile Vision Transformers. arXiv.
20. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
21. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Li, F. (2009, January 20–25). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
22. Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., and Guo, B. (2022, January 18–24). CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01181 ↗
23. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 10–17). Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
24. Wang, W., Yao, L., Chen, L., Cai, D., He, X., and Liu, W. (2021). Crossformer: A versatile vision transformer based on cross-scale attention. arXiv.
25. Lee, Y., Kim, J., Willette, J., and Hwang, S.J. (2022, January 18–24). Mpvit: Multi-path vision transformer for dense prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00714 ↗
26. Ren, S., Zhou, D., He, S., Feng, J., and Wang, X. (2022, January 18–24). Shunted self-attention via multi-scale token aggregation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01058 ↗
27. Peng, Z., Huang, W., Gu, S., Xie, L., Wang, Y., Jiao, J., and Ye, Q. (2021, January 10–17). Conformer: Local features coupling global representations for visual recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00042 ↗
28. Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., and Liu, Z. (2022, January 18–24). Mobile-former: Bridging mobilenet and transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00520 ↗
29. Vitae: Vision transformer advanced by exploring intrinsic inductive bias
30. Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., and Yang, L. (2020, January 6–12). Big Bird: Transformers for Longer Sequences. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
31. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020, January 13–18). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
32. Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., and Kaiser, L. (2020). Rethinking attention with performers. arXiv.
33. Wang, S., Li, B.Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv.
34. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
35. Woo, S., Park, J., Lee, J.-Y., and Kweon, I.S. (2018, January 8–14). CBAM: Convolutional Block Attention Module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
36. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2020, January 13–19). ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01155 ↗
37. Hassani, A., Walton, S., Li, J., Li, S., and Shi, H. (2023, January 17–24). Neighborhood attention transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00599 ↗
38. Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers. arXiv.
39. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
40. Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Jin, X., Wang, A., and Feng, J. (2021). All Tokens Matter: Token Labeling for Training Better Vision Transformers. arXiv.
41. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
42. Tan, M., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
43. Pan, J., Bulat, A., Tan, F., Zhu, X., Dudziak, L., Li, H., Tzimiropoulos, G., and Martinez, B. (2022, January 23–27). EdgeViTs: Competing Light-Weight CNNs on Mobile Devices with Vision Transformers. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20083-0_18 ↗
44. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018, January 18–23). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
45. Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. (2022, January 18–24). MetaFormer Is Actually What You Need for Vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01055 ↗
46. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
47. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. (2017, January 21–26). Scene Parsing through ADE20K Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.544 ↗
48. Semantic Understanding of Scenes through the ADE20K Dataset10.1007/s11263-018-1140-0 ↗
49. Kirillov, A., Girshick, R., He, K., and Dollár, P. (2019, January 15–19). Panoptic Feature Pyramid Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00656 ↗