Toward understanding the effectiveness of attention mechanism
Source abstract
Abstract
Attention mechanism (AM) is a widely used method for improving the performance of convolutional neural networks (CNNs) on computer vision tasks. Despite its pervasiveness, we have a poor understanding of what its effectiveness stems from. It is popularly believed that its effectiveness stems from the visual attention explanation, i.e., attention weights indicate the importance of feature and AM advocates focusing on the important part of an input image rather than ingesting the entire input. However, we find only a weak consistency exists between the attention weights of features and their importance. We verify the feature map multiplication that brings about high-order non-linearity into CNNs is crucial for the effectiveness of AM. Furthermore, we show an essential impact of feature map multiplication on the learned surfaces of CNNs. With the high-order non-linearity, feature map multiplication plays a regularization role on CNNs, which makes the learned curves smoother and more stable in-between real samples (test/training samples in datasets). Thus, compared to vanilla CNNs, CNNs equipped with AM are more robust to noises and yield smaller model sensitivity scores, which is the reason for their better performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Adversarial Robustness in Machine Learning · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Beijing University of Posts and Communications , Beijing, China
Funding
National Natural Science Foundation of China · Beijing Key Laboratory of Work Safety Intelligent Monitoring, Beijing University of Posts and Telecommunications
References (38)
Show source-provided references
1. Human pose estimation and its application to action recognition: A survey10.1016/j.jvcir.2021.103055 ↗
2. Image quality assessment using deep convolutional networks10.1063/1.5010804 ↗
3. Deep learning methods for super-resolution reconstruction of temperature fields in a supersonic combustor10.1063/5.0030040 ↗
4. ImageNet classification with deep convolutional neural networks
5. Deep residual learning for image recognition10.1109/cvpr.2016.90 ↗
6. Transform-invariant convolutional neural networks for image classification and search
7. SSD: Single shot multibox detector
8. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
9. H. Law, Y. Teng, O. Russakovsky, and J. Deng, “CornerNet-Lite: Efficient keypoint based object detection,” arXiv:1904.08900 (2019).
10. Feature pyramid networks for object detection
11. M2det: A single-shot object detector based on multi-level feature pyramid network
12. Fully convolutional networks for semantic segmentation
13. L. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv:1706.05587 (2017).
14. Batch normalization: Accelerating deep network training by reducing internal covariate shift
15. Going deeper with convolutions
16. Residual attention network for image classification
17. O. Oktay, J. Schlemper, L. L. Folgoc, M. C. H. Lee, M. P. Heinrich, K. Misawa, K. Mori, S. G. McDonagh, N. Y. Hammerla, B. Kainz, B. Glocker, and D. Rueckert, “Attention U-Net: Learning where to look for the pancreas,” arXiv:1804.03999 (2018).
18. Squeeze-and-excitation networks10.1109/CVPR.2018.00745 ↗
19. Cbam: Convolutional block attention module
20. Bounding and counting linear regions of deep neural networks
21. Comparing and aggregating partial orders with Kendall Tau distances10.1142/s1793830913600033 ↗
22. Full-gradient representation for neural network visualization
23. D. Smilkov, N. Thorat, B. Kim, F. Vié gas, and M. Wattenberg, “SmoothGrad: Removing noise by adding noise,” arXiv:1706.03825 (2017).
24. The MNIST database of handwritten digit images for machine learning research10.1109/msp.2012.2211477 ↗
25. Learning multiple layers of features from tiny images
26. On the number of linear regions of deep neural networks
27. 2018
28. mixup: Beyond empirical risk minimization
29. Generalization comparison of deep neural networks via output sensitivity
30. Spatial transformer networks
31. 2018 · 267
32. ECA-Net: Efficient channel attention for deep convolutional neural networks
33. Concurrent spatial and channel ‘squeeze & excitation’in fully convolutional networks
34. Dynamic convolution: Attention over convolution kernels
35. CondConv: Conditionally parameterized convolutions for efficient inference
36. Y. Zhang, J. Zhang, Q. Wang, and Z. Zhong, “DyNet: Dynamic convolution for accelerating convolutional neural networks,” arXiv:2004.10694 (2020).
37. Non-linear convolution filters for CNN-based learning
38. Kervolutional neural networks