Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
Source abstract
Abstract
This paper explores a simple CNN architecture used for image classification. Since the first introduction of the CNN idea, LeNet5, CNNs have become the main method for image exploration. A previously unexplored activation function is proposed, which improves the accuracy on one hand, while reducing the execution time. The other positive side is the faster convergence of the loss function. Unlike other well-known activation functions, such as ReLU, Tanh, and others, Exponential Partial Unit (EPU) does not overfit after 100 or more iterations. For this study, the CIFAR-10 dataset is chosen, which is a benchmark for this kind of research. This paper aims to present another view on CNNs, showing that effective networks can be trained with fewer parameters and reduced computational resources.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Medical Imaging Techniques and Applications · Brain Tumor Detection and Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Technology School Electronic Systems (TUES), associated with Technical University-Sofia, 1750 Sofia, Bulgaria
Independent Researcher, 1463 Sofia, Bulgaria
Faculty of Telecommunication, Technical University of Sofia, 1756 Sofia, Bulgaria
Funding
European Regional Development Fund
References (39)
Show source-provided references
1. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2014). Going Deeper with Convolutions. arXiv.10.1109/CVPR.2015.7298594 ↗
3. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗
4. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Li, F.-F. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
5. Coates, A., Lee, H., and Ng, A.Y. (2011, January 11–13). An Analysis of Single-Layer Networks in Unsupervised Feature Learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA.
6. Li, F.-F., Fergus, R., and Perona, P. (July, January 27). Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA.
7. LeCun, Y., Cortes, C., and Burges, C.J.C. (2025, April 10). The MNIST Database of Handwritten Digits. Available online: http://yann.lecun.com/exdb/mnist.
8. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. [Master’s Thesis, University of Tront].
9. Darlow, L.N., Crowley, E.J., Antoniou, A., and Storkey, A.J. (2018). CINIC-10 is not ImageNet or CIFAR-10. arXiv.
10. Doon, R., Kumar Rawat, T., and Gautam, S. (December, January 30). Cifar-10 Classification using Deep Convolutional Neural Network. Proceedings of the 2018 IEEE Punecon, Pune, India.10.1109/PUNECON.2018.8745428 ↗
11. Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A., and Bengio, Y. (2015). ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arXiv.
12. Calik, R.C., and Demirci, M.F. (November, January 28). Cifar-10 Image Classification with Convolutional Neural Networks for Embedded Systems. Proceedings of the 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), Aqaba, Jordan.10.1109/AICCSA.2018.8612873 ↗
13. PowerPack: Energy Profiling and Analysis of High-Performance Systems and Applications10.1109/TPDS.2009.76 ↗
14. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv.
15. Pathak, D., and El-Sharkawy, M. (2018, January 13–15). ReducedSqNet: A Shallow Architecture for CIFAR-10. Proceedings of the 2018 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA.10.1109/CSCI46756.2018.00078 ↗
16. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
17. Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16–21). Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
18. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2016). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv.
19. Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). Self-Normalizing Neural Networks. arXiv.
20. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv.10.1109/ICCV.2015.123 ↗
21. Maniatopoulos, A., and Mitianoudis, N. (2021). Learnable Leaky ReLU (LeLeLU): An Alternative Accuracy-Optimized Activation Function. Information, 12.10.3390/info12120513 ↗
22. Mahima, R., Maheswari, M., Roshana, S., Priyanka, E., Mohanan, N., and Nandhini, N. (2023, January 6–8). A Comparative Analysis of the Most Commonly Used Activation Functions in Deep Neural Network. Proceedings of the 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India.10.1109/ICESC57686.2023.10193390 ↗
23. Kaloev, M., and Krastev, G. (2021, January 11–13). Comparative Analysis of Activation Functions Used in the Hidden Layers of Deep Neural Networks. Proceedings of the 2021 3rd International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), Ankara, Turkey.10.1109/HORA52670.2021.9461312 ↗
24. Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv.
25. Wu, Y., and He, K. (2018). Group Normalization. arXiv.10.1007/978-3-030-01261-8_1 ↗
26. Luo, P., Ren, J., Peng, Z., Zhang, R., and Li, J. (2019). Differentiable Learning-to-Normalize via Switchable Normalization. arXiv.
27. Cortes, C., Mohri, M., and Rostamizadeh, A. (2012). L2 Regularization for Learning Kernels. arXiv.
28. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
29. Phaisangittisagul, E. (2016, January 25–27). An Analysis of the Regularization Between L2 and Dropout in Single Hidden Layer Neural Network. Proceedings of the 2016 7th International Conference on Intelligent Systems, Modelling and Simulation (ISMS), Bangkok, Thailand.10.1109/ISMS.2016.14 ↗
30. Veeramacheneni, L., Wolter, M., Klein, R., and Garcke, J. (2022, January 18–23). Canonical convolutional neural networks. Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy.10.1109/IJCNN55064.2022.9892607 ↗
31. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy.
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
33. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
34. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.10.1007/978-3-319-24574-4_28 ↗
35. Recognition of handwritten Katakana in a frame using moment invariants based on neural network10.1109/IJCNN.1991.170475 ↗
36. Telore, A.V., and Parashar, D. (2018, January 10–12). Detection and Rectification of Distorted Fingerprints Using Geometric Features and FFNN. Proceedings of the 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Bangalore, India.10.1109/ICCCNT.2018.8494089 ↗
37. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
38. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
39. Chen, C., Yang, Y., Xiang, Y., and Hao, W. (2025). Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations. arXiv.