Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
Source abstract
Abstract
Bilinear pooling, as an aggregation approach that outputs second-order statistics of deep learning features, has demonstrated effectiveness in a wide range of visual recognition tasks. Among major improvements on the bilinear pooling, matrix square root normalization—applied to the bilinear representation matrix—is regarded as a crucial step for further boosting performance. However, most existing works leverage Newton’s iteration to perform normalization, which becomes computationally inefficient when dealing with high-dimensional features. To address this limitation, through a comprehensive analysis, we reveal that both the distribution and magnitude of eigenvalues in the bilinear representation matrix play an important role in the network performance. Building upon this insight, we propose a novel approach, namely RegCov, which regularizes the eigenvalues when the normalization is absent. Specifically, RegCov incorporates two regularization terms that encourage the network to align the current eigenvalues with the target ones in terms of their distribution and magnitude. We implement RegCov across different network architectures and run extensive experiments on the ImageNet1K and fine-grained image classification benchmarks. The results demonstrate that RegCov maintains robust recognition to diverse datasets and network architectures while achieving superior inference speed compared to previous works.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sparse and Compressive Sensing Techniques · Advanced Data Compression Techniques · Medical Image Segmentation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an 710055, China
Funding
Natural Science Basic Research Program of Shaanxi · National Natural Science Foundation of China
References (74)
Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
2. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
3. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
4. Zhang, S., Murray, N., Wang, L., and Koniusz, P. (2022, January 23–27). Time-reversed diffusion tensor transformer: A new tenet of few-shot object detection. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20044-1_18 ↗
5. Zhang, S., Wang, L., Murray, N., and Koniusz, P. (2022, January 18–24). Kernelized few-shot object detection with efficient integral aggregation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01861 ↗
6. Wang, L., and Koniusz, P. (2023, January 17–24). 3mformer: Multi-order multi-mode transformer for skeletal action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00544 ↗
7. Multi-head second-order pooling for graph transformer networks10.1016/j.patrec.2023.01.017 ↗
8. Rahman, S., Koniusz, P., Wang, L., Zhou, L., Moghadam, P., and Sun, C. (2023, January 17–24). Learning partial correlation based deep visual representation for image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00603 ↗
9. Improved Bilinear Pooling With Pseudo Square-Rooted Matrix10.1109/LSP.2023.3262059 ↗
10. DropCov: A simple yet effective method for improving deep architectures
11. Song, Y., Sebe, N., and Wang, W. (2022, January 23–27). Batch-efficient eigendecomposition for small and medium matrices. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20050-2_34 ↗
12. On the Eigenvalues of Global Covariance Pooling for Fine-Grained Visual Recognition
13. Bilinear convolutional neural networks for fine-grained visual recognition10.1109/TPAMI.2017.2723400 ↗
14. Perronnin, F., Sánchez, J., and Mensink, T. (2010, January 5–11). Improving the fisher kernel for large-scale image classification. Proceedings of the Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece. Part IV 11.10.1007/978-3-642-15561-1_11 ↗
15. Kim, T.-K., Stefanos Zafeiriou, G.B., and Mikolajczyk, K. (2017, January 4–7). Improved Bilinear Pooling with CNNs. Proceedings of the British Machine Vision Conference (BMVC), London, UK.
16. Gao, Y., Beijbom, O., Zhang, N., and Darrell, T. (2016, January 27–30). Compact bilinear pooling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.41 ↗
17. Gou, M., Xiong, F., Camps, O., and Sznaier, M. (2018, January 18–23). Monet: Moments embedding network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00335 ↗
18. Carreira, J., Caseiro, R., Batista, J., and Sminchisescu, C. (2012, January 7–13). Semantic segmentation with second-order pooling. Proceedings of the Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy. Part VII 12.10.1007/978-3-642-33786-4_32 ↗
19. Ionescu, C., Vantzos, O., and Sminchisescu, C. (2015, January 7–13). Matrix backpropagation for deep networks with structured layers. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.339 ↗
20. Li, P., Xie, J., Wang, Q., and Gao, Z. (2018, January 18–22). Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00105 ↗
21. Deep CNNs meet global covariance pooling: Better representation and generalization
22. Non-Euclidean statistics for covariance matrices, with applications to diffusion tensor imaging10.1214/09-AOAS249 ↗
23. Yu, T., Cai, Y., and Li, P. (2020, January 23–28). Toward faster and simpler matrix normalization via rank-1 update. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Part XIX 16.10.1007/978-3-030-58529-7_13 ↗
24. Towards A Deeper Understanding of Global Covariance Pooling in Deep Learning: An Optimization Perspective10.1109/TPAMI.2023.3321392 ↗
25. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2021, October 26). The Caltech-Ucsd Birds-200-2011 Dataset. Available online: https://www.vision.caltech.edu/datasets/cub_200_2011/.
26. Dropout: A simple way to prevent neural networks from overfitting
27. Wang, S., and Manning, C. (2013, January 17–19). Fast dropout training. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA.
28. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
29. Separating style and content with bilinear models10.1162/089976600300015349 ↗
30. Yu, T., Li, X., and Li, P. (2021, January 2–9). Fast and compact bilinear pooling by shifted random Maclaurin. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i4.16435 ↗
31. Yu, T., Cai, Y., and Li, P. (March, January 22). Efficient compact bilinear pooling via Kronecker product. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.
32. GBP: Graph convolutional network embedded in bilinear pooling for fine-grained encoding10.1016/j.compeleceng.2024.109158 ↗
33. Category-aware multimodal attention network for fashion compatibility modeling10.1109/TMM.2023.3246796 ↗
34. Audiovisual dependency attention for violence detection in videos10.1109/TMM.2022.3184533 ↗
35. Koniusz, P., Zhang, H., and Porikli, F. (2018, January 18–23). A deeper look at power normalizations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00605 ↗
36. Normalization techniques in training dnns: Methodology, analysis and application10.1109/TPAMI.2023.3250241 ↗
37. Lee, S., Bae, J., and Kim, H.Y. (2023, January 17–24). Decompose, adjust, compose: Effective normalization by playing with frequency for domain generalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01133 ↗
38. Lei, J., Hu, X., Wang, Y., and Liu, D. (2023, January 17–24). Pyramidflow: High-resolution defect contrastive localization using pyramid normalizing flow. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01359 ↗
39. Deepnet: Scaling transformers to 1,000 layers10.1109/TPAMI.2024.3386927 ↗
40. Batch normalization embeddings for deep domain generalization10.1016/j.patcog.2022.109115 ↗
41. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
42. Ni, Y., Guo, Y., Jia, J., and Huang, L. (2024, January 21–27). On the nonlinearity of layer normalization. Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria.
43. Arsigny, V., Fillard, P., Pennec, X., and Ayache, N. (2005, January 26–29). Fast and simple calculus on tensors in the log-Euclidean framework. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Palm Springs, CA, USA.10.1007/11566465_15 ↗
44. Adaptive Log-Euclidean metrics for SPD matrix learning10.1109/TIP.2024.3451930 ↗
45. Huang, Z., and Van Gool, L. (2017, January 4–9). A riemannian network for spd matrix learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10866 ↗
46. Wei, X., Zhang, Y., Gong, Y., Zhang, J., and Zheng, N. (2018, January 8–14). Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_22 ↗
47. Huang, L., Yang, D., Lang, B., and Deng, J. (2018, January 18–22). Decorrelated batch normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00089 ↗
48. Huang, L., Zhou, Y., Zhu, F., Liu, L., and Shao, L. (2019, January 16–17). Iterative normalization: Beyond standardization towards efficient whitening. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00501 ↗
49. Understanding whitening loss in self-supervised learning10.1109/TPAMI.2024.3417438 ↗
50. Mukuta, Y., Machida, T., and Harada, T. (2019). Compact approximation for polynomial of covariance feature. arXiv.
51. Fast Differentiable Matrix Square Root and Inverse Square Root10.1109/TPAMI.2022.3216339 ↗
52. Engin, M., Wang, L., Zhou, L., and Liu, X. (2018, January 8–14). Deepkspd: Learning kernel-matrix-based spd representation for fine-grained image recognition. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_38 ↗
53. Rahman, S., Wang, L., Sun, C., and Zhou, L. (2020, January 23–28). Redro: Efficiently learning large-sized spd visual representation. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58555-6_1 ↗
54. Lin, T.Y., Maji, S., and Koniusz, P. (2018, January 8–14). Second-order democratic aggregation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_38 ↗
55. Chen, Z., Song, Y., Liu, G., Kompella, R.R., Wu, X.J., and Sebe, N. (2024, January 16–22). Riemannian Multinomial Logistics Regression for SPD Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01617 ↗
56. Bazzi, A., Slock, D.T., and Meilhac, L. (2016, January 20–25). Detection of the number of superimposed signals using modified MDL criterion: A random matrix approach. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472547 ↗
57. Li, P., Xie, J., Wang, Q., and Zuo, W. (2017, January 22–29). Is second-order information helpful for large-scale visual recognition?. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.228 ↗
58. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
59. Yang, Z., Zeng, A., Li, Z., Zhang, T., Yuan, C., and Li, Y. (2023, January 2–3). From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01576 ↗
60. Dong, J., Koniusz, P., Chen, J., Wang, Z.J., and Ong, Y.S. (2024, January 16–22). Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02686 ↗
61. Chen, D., Mei, J.P., Zhang, H., Wang, C., Feng, Y., and Chen, C. (2022, January 18–24). Knowledge distillation with the reused teacher classifier. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01163 ↗
62. Miles, R., and Mikolajczyk, K. (2024, January 20–27). Understanding the role of the projector in knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
63. Yang, Z., Li, Z., Shao, M., Shi, D., Yuan, Z., and Yuan, C. (2022, January 23–27). Masked generative distillation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20083-0_4 ↗
64. Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., and Yuan, C. (2022, January 18–24). Focal and global knowledge distillation for detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00460 ↗
65. Li, L. (2022, January 23–27). Self-regulated feature learning via teacher-free feature distillation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19809-0_20 ↗
66. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. (2013). Fine-grained visual classification of aircraft. arXiv.
67. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 2–8). 3d object representations for fine-grained categorization. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia.10.1109/ICCVW.2013.77 ↗
68. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
69. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
70. Chen, Z., Song, Y., Wu, X., Liu, G., and Sebe, N. (2025, January 24). Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry. Proceedings of the Thirteenth International Conference on Learning Representations, Singapore.
71. Yang, Y., and Newsam, S. (2010, January 3–5). Bag-of-visual-words and spatial extensions for land-use classification. Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, San Jose, CA, USA.10.1145/1869790.1869829 ↗
72. Improved bilinear CNN model for remote sensing scene classification
73. Song, Y., Sebe, N., and Wang, W. (2021, January 11–17). Why approximate matrix square root outperforms accurate SVD in global covariance pooling?. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00115 ↗
74. Backpropagation-friendly eigendecomposition