A Comparison of Pooling Methods for Convolutional Neural Networks
Source abstract
Abstract
One of the most promising techniques used in various sciences is deep neural networks (DNNs). A special type of DNN called a convolutional neural network (CNN) consists of several convolutional layers, each preceded by an activation function and a pooling layer. The feature map of the previous layer is sampled by the pooling layer (that seems to be an important layer) to create a new feature map with condensed resolution. This layer significantly reduces the spatial dimension of the input. It always accomplished two main goals. As a first step, it reduces the number of parameters or weights to minimize computational costs. The second step is to prevent the overfitting of the network. In addition, pooling techniques can significantly reduce model training time and computational costs. This paper provides a critical understanding of traditional and modern pooling techniques and highlights the strengths and weaknesses for readers. Moreover, the performance of pooling techniques on different datasets is qualitatively evaluated and reviewed. This study is expected to contribute to a comprehensive understanding of the importance of CNNs and pooling techniques in computer vision challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Anomaly Detection Techniques and Applications · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, Parit Raja 86400, Malaysia
School of Electronics, Computing, and Mathematics, University of Derby, Derby DE22 1GB, UK
Department of Computer Science, National University of Technology, Islamabad 44000, Pakistan
Department of Information Technology, College of Computer and Information Sciences, Majmaah University, Al Majmaah 11952, Saudi Arabia
Department of Computer Science and Information Systems, College of Applied Sciences, AlMaarefa University, Riyadh 13713, Saudi Arabia
Department of Natural and Applied Sciences, Faculty of Community College, Majmaah University, Majmaah 11952, Saudi Arabia
Funding
Majmaah University · Almaarefa University
References (79)
Show source-provided references
1. A machine learning method to estimate PM2.5 concentrations across China with remote sensing, meteorological and land use information10.1016/j.scitotenv.2018.04.251 ↗
2. Learning pattern classification-a survey10.1109/18.720536 ↗
3. Principal components, minor components, and linear neural networks10.1016/S0893-6080(05)80089-9 ↗
4. Ellacott, S.W. (1990, January 9–13). An analysis of the delta rule. Proceedings of the International Neural Network Conference, Paris, France.10.1007/978-94-009-0643-3_145 ↗
5. Learning representations by back-propagating errors10.1038/323533a0 ↗
6. Mehdipour, G.M., and Kemal, E.H. (2016, January 27–30). A comprehensive analysis of deep learning based representation for face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA.
7. Nagpal, S., Singh, M., Vatsa, M., and Singh, R. (2015, January 8–11). Regularizing deep learning architecture for face recognition with weight variations. Proceedings of the 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), Arlington, VA, USA.10.1109/BTAS.2015.7358791 ↗
8. Backpropagation applied to handwritten zip code recognition10.1162/neco.1989.1.4.541 ↗
9. Deep convolution neural network for image recognition10.1016/j.ecoinf.2018.10.002 ↗
10. InceptB: A CNN based classification approach for recognizing traditional bengali games10.1016/j.procs.2018.10.436 ↗
11. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
12. Siddique, F., Sakib, S., and Siddique, M.A. (2019, January 26–28). Recognition of handwritten digit using convolutional neural network in python with tensorflow and comparison of performance for various hidden layers. Proceedings of the 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), Dhaka, Bangladesh.10.1109/ICAEE48663.2019.8975496 ↗
13. Deep learning for AI10.1145/3448250 ↗
14. Yu, D., Wang, H., Chen, P., and Wei, Z. (2014, January 24–26). Mixed pooling for convolutional neural networks. Proceedings of the International Conference on Rough Sets and Knowledge Technology, Shanghai, China.10.1007/978-3-319-11740-9_34 ↗
15. Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. (2019). Once-for-all: Train one network and specialize it for efficient deployment. arXiv.
16. A new approach for arrhythmia classification using deep coded features and LSTM networks10.1016/j.cmpb.2019.05.004 ↗
17. Zhao, R., Song, W., Zhang, W., Xing, T., Lin, J.H., Srivastava, M., Gupta, R., and Zhang, Z. (2017, January 22–24). Accelerating binarized convolutional neural networks with software-programmable FPGAs. Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA.10.1145/3020078.3021741 ↗
18. Murray, N., and Perronnin, F. (2014, January 23–28). Generalized max pooling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.317 ↗
19. Towards dropout training for convolutional neural networks10.1016/j.neunet.2015.07.007 ↗
20. Boureau, Y.L., Ponce, J., and LeCun, Y. (2010, January 21–24). A theoretical analysis of feature pooling in visual recognition. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
21. Ensemble transfer CNNs driven by multi-channel signals for fault diagnosis of rotating machinery cross working conditions10.1016/j.knosys.2020.106396 ↗
22. Hybrid MPSO-CNN: Multi-level particle swarm optimized hyperparameters of convolutional neural network10.1016/j.swevo.2021.100863 ↗
23. A comparative analysis of pooling strategies for convolutional neural network based Hindi ASR10.1007/s12652-019-01325-y ↗
24. Prediction of ball milling performance by a convolutional neural network model and transfer learning10.1016/j.powtec.2022.117409 ↗
25. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
26. A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load10.1016/j.ymssp.2017.06.022 ↗
27. Stergiou, A., Poppe, R., and Kalliatakis, G. (2021, January 11–17). Refining activation downsampling with SoftPool. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01019 ↗
28. Mixed pooling and richer attention feature fusion for crack detection10.1016/j.patrec.2021.02.005 ↗
29. Automated diagnosis of multi-class brain abnormalities using MRI images: A deep convolutional neural network based method10.1016/j.patrec.2020.04.018 ↗
30. Deliège, A., Istasse, M., Kumar, A., De Vleeschouwer, C., and Van Droogenbroeck, M. (2021). Ordinal pooling. arXiv.
31. Sharma, T., Verma, N.K., and Masood, S. (2022). Mixed fuzzy pooling in convolutional neural networks for image classification. Multimed. Tools Appl., 1–7.10.1007/s11042-022-13553-0 ↗
32. Lee, C.Y., Gallagher, P.W., and Tu, Z. (2016, January 9–11). Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, Cadiz, Spain.
33. Improved audio scene classification based on label-tree embeddings and convolutional neural networks10.1109/TASLP.2017.2690564 ↗
34. Deep neural network to extract high-level features and labels in multi-label classification problems10.1016/j.neucom.2020.06.117 ↗
35. Linking functional traits to multiscale statistics of leaf venation networks10.1111/nph.16830 ↗
36. Zeiler, M.D., and Fergus, R. (2013). Stochastic pooling for regularization of deep convolutional neural networks. arXiv.
37. Rank-based pooling for deep convolutional neural networks10.1016/j.neunet.2016.07.003 ↗
38. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
39. Medical image analysis using convolutional neural networks: A review10.1007/s10916-018-1088-1 ↗
40. Ni, R., Goldblum, M., Sharaf, A., Kong, K., and Goldstein, T. (2021, January 18–24). Data augmentation for meta-learning. Proceedings of the International Conference on Machine Learning (PMLR), Virtual Event.
41. Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs10.1016/j.neucom.2018.03.080 ↗
42. Superpixel based land cover classification of VHR satellite image combining multi-scale CNN and scale parameter estimation10.1007/s12145-019-00383-2 ↗
43. A novel stochastic resonance model based on bistable stochastic pooling network and its application10.1016/j.chaos.2021.110800 ↗
44. Grauman, K., and Darrell, T. (2005, January 17–21). The pyramid match kernel: Discriminative classification with sets of image features. Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), Beijing, China.10.1109/ICCV.2005.239 ↗
45. Spatial pyramid pooling in deep convolutional networks for visual recognition10.1109/TPAMI.2015.2389824 ↗
46. Bekkers, E.J. (2019). B-spline cnns on lie groups. arXiv.
47. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
48. Data-driven based tiny-YOLOv3 method for front vehicle detection inducing SPP-net10.1109/ACCESS.2020.3001279 ↗
49. Computer vision-based approach for smart traffic condition assessment at the railroad grade crossing10.1016/j.aei.2021.101456 ↗
50. CNN architectures for geometric transformation-invariant feature representation in computer vision: A review10.1007/s42979-021-00735-0 ↗
51. Rapid detection of blind roads and crosswalks by using a lightweight semantic segmentation network10.1109/TITS.2020.2989129 ↗
52. Yu, T., Li, X., and Li, P. (2021, January 2–9). Fast and compact bilinear pooling by shifted random Maclaurin. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v35i4.16435 ↗
53. No-reference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling10.1016/j.patcog.2019.107174 ↗
54. Rippel, O., Snoek, J., and Adams, R.P. (2015). Spectral representations for convolutional neural networks. Adv. Neural Inf. Process. Syst., 28.
55. Revaud, J., Leroy, V., Weinzaepfel, P., and Chidlovskii, B. (2022, January 19–23). PUMP: Pyramidal and Uniqueness Matching Priors for Unsupervised Learning of Local Descriptors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00390 ↗
56. Effect of pooling strategy on convolutional neural network for classification of hyperspectral remote sensing images10.1049/iet-ipr.2019.0561 ↗
57. Graham, B. (2014). Fractional max-pooling. arXiv.
58. Zhai, S., Wu, H., Kumar, A., Cheng, Y., Lu, Y., Zhang, Z., and Feris, R. (, January 21–26). S3pool: Pooling with stochastic spatial sampling. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.426 ↗
59. Machine-learning-aided optical fiber communication system10.1109/MNET.011.2000676 ↗
60. A high-precision detection method of hydroponic lettuce seedlings status based on improved Faster RCNN10.1016/j.compag.2021.106054 ↗
61. Saeedan, F., Weber, N., Goesele, M., and Roth, S. (2018, January 18–23). Detail-preserving pooling in deep networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018, Salt Lake City, UT, USA.10.1109/CVPR.2018.00949 ↗
62. Gao, Z., Wang, L., and Wu, G. (November, January 27). Lip: Local importance-based pooling. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00345 ↗
63. RNNPool: Efficient non-linear pooling for RAM constrained inference
64. Chen, Y., Liu, Z., and Shi, Y. (2021, January 25–28). RP-Unet: A Unet-based network with RNNPool enables computation-efficient polyp segmentation. Proceedings of the Sixth International Workshop on Pattern Recognition, Beijing, China.10.1117/12.2604803 ↗
65. VISPNN: VGG-inspired stochastic pooling neural network
66. Benkaddour, M.K. (2021). CNN based features extraction for age estimation and gender classification. Informatica, 45.10.31449/inf.v45i5.3262 ↗
67. Interpretation of intelligence in CNN-pooling processes: A methodological survey10.1007/s00521-019-04296-5 ↗
68. Lee, D., Lee, S., and Yu, H. (2021, January 2–9). Learnable dynamic temporal pooling for time series classification. Proceedings of the AAAI Conference on Artificial Intelligence 2021, Vancouver, BC, Canada.10.1609/aaai.v35i9.17008 ↗
69. Zhang, H., and Ma, J. (2018). Hartley spectral pooling for deep learning. arXiv.
70. Li, H., Ouyang, W., and Wang, X. (2016, January 19–24). Multi-bias non-linear activation in deep neural networks. Proceedings of the International Conference on Machine Learning 2016, New York City, NY, USA.
71. Williams, T., and Li, R. (May, January 30). Wavelet pooling for convolutional neural networks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
72. Combining max-pooling and wavelet pooling strategies for semantic image segmentation10.1016/j.eswa.2021.115403 ↗
73. Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017, January 14–19). EMNIST: Extending MNIST to handwritten letters. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.10.1109/IJCNN.2017.7966217 ↗
74. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2018). Do cifar-10 classifiers generalize to cifar-10?. arXiv.
75. An analysis of convolutional neural networks for image classification10.1016/j.procs.2018.05.198 ↗
76. Multi-class brain tumor classification using residual network and global average pooling10.1007/s11042-020-10335-4 ↗
77. An image-level weakly supervised segmentation method for No-service rail surface defect with size prior10.1016/j.ymssp.2021.108334 ↗
78. Santos, C.F.G.D., and Papa, J.P. (2022). Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Comput. Surv. (CSUR).10.1145/3510413 ↗
79. Encoding spatial distribution of convolutional features for texture representation