φunit: A Lightweight Module for Feature Fusion Based on Their Dimensions
Source abstract
Abstract
With the popularity of mobile devices, lightweight deep learning models have important value in various application scenarios. However, how to effectively fuse the feature information from different dimensions while ensuring the model’s lightness and high accuracy is a problem that has not been fully solved. In this paper, we propose a novel feature fusion module, called φunit, which can fuse the features extracted by different dimensional networks according to the order of feature information with a small computational cost, avoiding the problems of information fragmentation caused by simple feature stacking in traditional information fusion. Based on φunit, this paper further builds an extremely lightweight model φNet, which can achieve performance close to the highest accuracy on several public datasets under the condition of very limited parameter scale. The core idea of φunit is to use deconvolution to reduce the discrepancy among the features to be fused, and to lower the possibility of feature information fragmentation after fusion by fusing the features from different dimensions sequentially. φNet is a lightweight network composed of multiple φunits and bottleneck modules, with a parameter scale of only 1.24 M, much smaller than traditional lightweight models. This paper conducts experiments on public datasets, and φNet achieves an accuracy of 71.64% on the food101 dataset, and an accuracy of 75.31% on the random 50-category food101 dataset, both higher than or close to the highest accuracy. This paper provides a new idea and method for feature fusion of lightweight models, and also provides an efficient model selection for deep learning applications on mobile devices.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China
Research Center of Intelligent Information Processing and Quantum Intelligent Computing, Shanghai 201306, China
Funding
National Key R&D Plan
References (54)
Show source-provided references
1. Zhang, X., Li, S., Li, X., Huang, P., Shan, J., and Chen, T. (2023, January 18–22). DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00381 ↗
2. Liu, Z., Zhou, Y., Xu, Y., and Wang, Z. (2023, January 18–22). Simplenet: A simple network for image anomaly detection and localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01954 ↗
3. Jiang, H., Dang, Z., Wei, Z., Xie, J., Yang, J., and Salzmann, M. (2023, January 18–22). Robust Outlier Rejection for 3D Registration with Variational Bayes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00117 ↗
4. Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., and Stiefelhagen, R. (2023, January 18–22). Delivering Arbitrary-Modal Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00116 ↗
5. Ru, L., Zheng, H., Zhan, Y., and Du, B. (2023, January 18–22). Token contrast for weakly-supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00302 ↗
6. Jiang, J., and Zheng, N. (2023, January 18–22). MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02318 ↗
7. Yue, H., Wang, K., Zhang, G., Feng, H., Han, J., Ding, E., and Wang, J. (2023, January 7–14). Cyclically disentangled feature translation for face anti-spoofing. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i3.25443 ↗
8. Liu, C., Yu, X., Tsai, Y.H., Faraki, M., Moslemi, R., Chandraker, M., and Fu, Y. (2022, January 19–24). Learning to learn across diverse data biases in deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00404 ↗
9. Wang, K., Wang, S., Zhang, P., Zhou, Z., Zhu, Z., Wang, X., Peng, X., Sun, B., Li, H., and You, Y. (2022, January 19–24). An efficient training approach for very large scale face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00405 ↗
10. Tang, Z., Qiu, Z., Hao, Y., Hong, R., and Yao, T. (2023, January 18–22). 3D Human Pose Estimation with Spatio-Temporal Criss-Cross Attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00464 ↗
11. Wang, Z., Nie, X., Qu, X., Chen, Y., and Liu, S. (2022, January 19–24). Distribution-aware single-stage models for multi-person 3D pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01275 ↗
12. Weng, C.Y., Curless, B., Srinivasan, P.P., Barron, J.T., and Kemelmacher-Shlizerman, I. (2022, January 19–24). Humannerf: Free-viewpoint rendering of moving people from monocular video. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01573 ↗
13. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
14. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
15. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
16. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
17. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
18. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
19. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
20. Zagoruyko, S., and Komodakis, N. (2016). Wide residual networks. arXiv.10.5244/C.30.87 ↗
21. Han, D., Kim, J., and Kim, J. (2017, January 21–26). Deep pyramidal residual networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.668 ↗
22. Mahendran, N. (2023). Squeeze aggregated excitation network. arXiv.
23. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
24. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and <0.5 MB model size. arXiv.
25. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
26. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
27. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
28. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2020, January 14–19). ECA-Net: Efficient channel attention for deep convolutional neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01155 ↗
29. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
30. Han, S., Mao, H., and Dally, W.J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv.
31. Fang, G., Ma, X., Song, M., Mi, M.B., and Wang, X. (2023, January 18–22). Depgraph: Towards any structural pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01544 ↗
32. Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., and Shao, L. (2020, January 14–19). Hrank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00160 ↗
33. Pruning filter in filter
34. Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, J. (2022, January 19–24). Decoupled knowledge distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01165 ↗
35. Lin, S., Xie, H., Wang, B., Yu, K., Chang, X., Liang, X., and Wang, G. (2022, January 19–24). Knowledge distillation via the target-aware transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01064 ↗
36. Dong, P., Li, L., and Wei, Z. (2023, January 18–22). Diswot: Student architecture search for distillation without training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01145 ↗
37. Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. (2021, January 19–25). RepVGG: Making VGG-style ConvNets Great again. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01352 ↗
38. Choi, K., Lee, H.Y., Hong, D., Yu, J., Park, N., Kim, Y., and Lee, J. (2022, January 19–24). It’s All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00813 ↗
39. Yamamoto, K. (2021, January 19–25). Learnable companding quantization for accurate low-bit neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00499 ↗
40. Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., and Xu, C. (2020, January 14–19). Ghostnet: More features from cheap operations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00165 ↗
41. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
42. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18–22). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00716 ↗
43. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for mobilenetv3. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, South Korea.
44. GhostNetv2: Enhance cheap operation with long-range attention
45. Resmlp: Feedforward networks for image classification with data-efficient training10.1109/TPAMI.2022.3206148 ↗
46. Park, N., and Kim, S. (2022). How do vision transformers work?. arXiv.
47. Zeiler, M.D., Krishnan, D., Taylor, G.W., and Fergus, R. (2010, January 13–18). Deconvolutional networks. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5539957 ↗
48. Dumoulin, V., and Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv.
49. Image super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
50. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
51. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
52. Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., and Keutzer, K. (2018, January 18–22). Shift: A zero flop, zero parameter alternative to spatial convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00951 ↗
53. Loshchilov, I., and Hutter, F. (May, January 26). Fixing Weight Decay Regularization in Adam. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
54. Automatic diagnosis of Covid-19 related pneumonia from CXR and CT-Scan images10.48084/etasr.4613 ↗