Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network
Source abstract
Abstract
Light-weight convolutional neural networks (CNNs) suffer limited feature representation capabilities due to low computational budgets, resulting in degradation in performance. To make CNNs more efficient, dynamic neural networks (DyNet) have been proposed to increase the complexity of the model by using the Squeeze-and-Excitation (SE) module to adaptively obtain the importance of each convolution kernel through the attention mechanism. However, the attention mechanism in the SE network (SENet) selects all channel information for calculations, which brings essential challenges: (a) interference caused by the internal redundant information; and (b) increasing number of network calculations. To address the above problems, this work proposes a dynamic convolutional network (termed as EAM-DyNet) to reduce the number of channels in feature maps by extracting only the useful spatial information. EAM-DyNet first uses the random channel reduction and channel grouping reduction methods to remove the redundancy in the information. As the downsampling of information can lead to the loss of useful information, it then applies an adaptive average pooling method to maintain the information integrity. Extensive experimental results on the baseline demonstrate that EAM-DyNet outperformed the existing approaches, thus it can achieve higher accuracy of the network test and less network parameters.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221008, China
IOT Perception Mine Research Center, China University of Mining and Technology, Xuzhou 221008, China
Institute of Electrodynamics and Microelectronics, University of Bremen, 28359 Bremen, Germany
Funding
Research on Online Coal Gangue Recognition Method Based on Terahertz Detection Technology · National Key Research and Development Plan of China
References (44)
Show source-provided references
1. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
2. EPKF: Energy efficient communication schemes based on Kalman filter for IoT10.1109/JIOT.2019.2900853 ↗
3. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
4. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
5. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00140 ↗
6. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv.
7. Sandler, M., Howard, A., Zhu, M.L., Zhmoginov, A., and Chen, L.C. (2018, January 8–14). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision, Munich, Germany.10.1109/CVPR.2018.00474 ↗
8. Squeeze-and-excitation networks10.1109/TPAMI.2019.2913372 ↗
9. Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. (2018, January 8–14). ShuffleNet V2: Practical guidelines for efficient CNN architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_8 ↗
10. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 8–14). ShuffleNet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Munich, Germany.10.1109/CVPR.2018.00716 ↗
11. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alche-Buc, F., Fox, E., and Garnett, R. (2019). CondConv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems 32, Proceedings of the Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada, 8–14 December 2019, NeurIPS.
12. Zhang, Y., Zhang, J., Wang, Q., and Zhong, Z. (2020). DyNet: Dynamic convolution for accelerating convolutional neural networks. arXiv.
13. Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., and Liu, Z. (2019). Dynamic convolution: Attention over convolution kernels. arXiv.10.1109/CVPR42600.2020.01104 ↗
14. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and <0.5 MB model size. arXiv.
15. Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., and Keutzer, K. (2017). Shift: A zero FLOP, zero parameter alternative to spatial convolutions. arXiv.10.1109/CVPR.2018.00951 ↗
16. Sifre, L., and Mallat, S. (2014). Rigid-motion scattering for texture classification. arXiv.
17. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
18. Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q.V. (2019, January 15–20). MnasNet: Platform-aware neural architecture search for mobile. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00293 ↗
19. Zoph, B., and Le, Q.V. (2016). Neural architecture search with reinforcement learning. arXiv.
20. Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., and Adam, H. (2018, January 8–14). NetAdapt: Platform-aware neural network adaptation for mobile applications. Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01249-6_18 ↗
21. Cai, H., Zhu, L., and Han, S. (2018). ProxylessNAS: Direct neural architecture search on target task and hardware. arXiv.
22. Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J. (2019). Single path one-shot neural architecture search with uniform sampling. arXiv.10.1007/978-3-030-58517-4_32 ↗
23. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18–23). Learning transferable architectures for scalable image recognition. Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00907 ↗
24. Real, E., Aggarwal, A., Huang, Y.P., and Le, Q.V. (27–1, January 27). Regularized evolution for image classifier architecture search. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-2019), Honolulu, HI, USA.
25. Liu, H., Simonyan, K., and Yang, Y. (2018). DARTS: Differentiable architecture search. arXiv.
26. Xie, S., Zheng, H., Liu, C., and Lin, L. (2018). SNAS: Stochastic neural architecture search. arXiv.
27. Haase, D., and Amthor, M. (2020, January 13–19). Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved MobileNets. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01461 ↗
28. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Runtime neural pruning. Advances in Neural Information Processing Systems 30, Proceedings of the Annual Conference on Neural Information Processing Systems 2017, NeurIPS 2017, Long Beach, CA, USA, 4–9 December 2017, NeurIPS.
29. Liu, L., and Deng, J. (2018, January 2–7). Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, LA, USA.10.1609/aaai.v32i1.11630 ↗
30. Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., and Gonzalez, J.E. (2018, January 8–14). SkipNet: Learning dynamic routing in convolutional networks. Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01261-8_25 ↗
31. Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L.S., Grauman, K., and Feris, R. (2018, January 18–23). BlockDrop: Dynamic inference paths in residual networks. Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00919 ↗
32. Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., and Weinberger, K.Q. (2017). Multi-scale dense networks for resource efficient image classification. arXiv.
33. Yu, J., and Huang, T. (November, January 27). Universally slimmable networks and improved training techniques. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00189 ↗
34. Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. (2019). Once-for-all: Train one network and specialize it for efficient deployment. arXiv.
35. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with deep reinforcement learning. arXiv.
36. Luong, M.-T., Pham, H., and Manning, C.D. (2015). Effective approaches to attention-based neural machine translation. arXiv.10.18653/v1/D15-1166 ↗
37. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
39. Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., and Xu, C. (2020, January 13–19). GhostNet: More features from cheap operations. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00165 ↗
40. Zhang, Q., Jiang, Z., Lu, Q., Han, J.n., Zeng, Z., Gao, S.-h., and Men, A. (2020). Split to be slim: An overlooked redundancy in vanilla convolution. arXiv.10.24963/ijcai.2020/442 ↗
41. Krizhevsky, A. (2012). Learning Multiple Layers of Features from Tiny Images. [Master’s Thesis, University of Toronto].
42. Zagoruyko, S., and Komodakis, N. (2016). Wide residual networks. arXiv.10.5244/C.30.87 ↗
43. ImageNet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
44. Szegedy, C., Wei, L., Yangqing, J., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗