On the Relative Impact of Optimizers on Convolutional Neural Networks with Varying Depth and Width for Image Classification
Source abstract
Abstract
The continued increase in computing resources is one key factor that is allowing deep learning researchers to scale, design and train new and complex convolutional neural network (CNN) architectures in terms of varying width, depth, or both width and depth to improve performance for a variety of problems. The contributions of this study include an uncovering of how different optimization algorithms impact CNN architectural setups with variations in width, depth, and both width/depth. Specifically in this study, three different CNN architectural setups in combination with nine different optimization algorithms—namely SGD vanilla, with momentum, and with Nesterov momentum, RMSProp, ADAM, ADAGrad, ADADelta, ADAMax, and NADAM—are trained and evaluated using three publicly available benchmark image classification datasets. Through extensive experimentation, we analyze the output predictions of the different optimizers with the CNN architectures using accuracy, convergence speed, and loss function as performance metrics. Findings based on the overall results obtained across the three image classification datasets show that ADAM and NADAM achieved superior performances with wider and deeper/wider setups, respectively, while ADADelta was the worst performer, especially with the deeper CNN architectural setup.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Machine Learning and Data Classification · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Federal University of Technology, Minna 920211, Nigeria
Department of Electrical Engineering Science, University of Johannesburg, Johannesburg 2092, South Africa
Digital Transformation Portfolio, Tshwane University of Technology, Pretoria 0183, South Africa
References (45)
Show source-provided references
1. Papamakarios, G. (2014). Comparison of Stochastic Optimization Algorithms. University of Edinburgh.
2. Hallen, R. (2022, September 20). A Study of Gradient-Based Algorithms. Available online: http://lup.lub.lu.se/student-papers/record/8904399.
3. Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv.
4. Shalev-Shwartz, S., Shamir, O., and Shammah, S. (2017, January 6–11). Failures of gradient-based deep learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
5. Tschoepe, M. (2019). Beyond SGD: Recent Improvements of Gradient Descent Methods. [Master’s Thesis, Technische Universität Kaiserslautern].
6. Zhang, M., Lucas, J., Ba, J., and Hinton, G.E. (2019). Lookahead optimizer: K steps forward, 1 step back. Advances in Neural Information Processing Systems, MIT.
7. Zhu, A., Meng, Y., and Zhang, C. (2017, January 2–4). An improved Adam Algorithm using look-ahead. Proceedings of the 2017 International Conference on Deep Learning Technologies, Chengdu, China.10.1145/3094243.3094249 ↗
8. Reddi, S.J., Kale, S., and Kumar, S. (2019). On the convergence of adam and beyond. arXiv.
9. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013, January 16–21). On the importance of initialization and momentum in deep learning. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
10. Schaul, T., Antonoglou, I., and Silver, D. (2013). Unit tests for stochastic optimization. arXiv.
11. Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. (2019). On the variance of the adaptive learning rate and beyond. arXiv.
12. Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1987). Learning Internal Representations by Error Propagation. Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, MITP.
13. Nielsen, M.A. (2015). Neural Networks and Deep Learning, Determination Press.
14. Dogo, E.M., Afolabi, O.J., Nwulu, N.I., Twala, B., and Aigbavboa, C.O. (2018, January 21–22). A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. Proceedings of the 2018 International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS), Belgaum, India.10.1109/CTEMS.2018.8769211 ↗
15. Joshua, A.O., Nelwamondo, F.V., and Mabuza-Hocquet, G. (2019, January 28–30). Segmentation of optic cup and disc for diagnosis of glaucoma on retinal fundus images. Proceedings of the 2019 Southern African Universities Power Engineering Conference/Robotics and Mechatronics/Pattern Recognition Association of South Africa (SAUPEC/RobMech/PRASA), Bloemfontein, South Africa.10.1109/RoboMech.2019.8704727 ↗
16. The use of U-Net lite and Extreme Gradient Boost (XGB) for glaucoma detection10.1109/ACCESS.2021.3068204 ↗
17. Torsional capacity evaluation of RC beams using an improved bird swarm algorithm optimised 2D convolutional neural network10.1016/j.engstruct.2022.115066 ↗
18. Vision-based concrete crack detection using a hybrid framework considering noise effect10.1016/j.jobe.2022.105246 ↗
19. Wilson, A.C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Processing Systems, MIT.
20. Reconciling modern machine-learning practice and the classical bias–variance trade-off10.1073/pnas.1903070116 ↗
21. Nichani, E., Radhakrishnan, A., and Uhler, C. (2021, January 18–24). Do deeper convolutional networks perform better?. Proceedings of the International Conference on Machine Learning, Online.
22. Nguyen, T., Raghu, M., and Kornblith, S. (2020). Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. arXiv.
23. Tan, M., and Le, Q. (2019, January 9–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
24. Hastie, T., Tibshirani, R., Friedman, J.H., and Friedman, J.H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer.10.1007/978-0-387-84858-7 ↗
25. Yang, Z., Yu, Y., You, C., Steinhardt, J., and Ma, Y. (2020, January 13–18). Rethinking bias-variance trade-off for generalization of neural networks. Proceedings of the International Conference on Machine Learning, Online.
26. Towards learning convolutions from scratch
27. Urban, G., Geras, K.J., Kahou, S.E., Aslan, O., Wang, S., Caruana, R., and Richardson, M. (2016). Do deep convolutional nets really need to be deep and convolutional?. arXiv.
28. Nguyen, Q., and Hein, M. (2018, January 10–15). Optimization landscape and expressivity of deep CNNs. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
29. Keskar, N.S., and Socher, R. (2017). Improving generalization performance by switching from adam to sgd. arXiv.
30. Choi, D., Shallue, C.J., Nado, Z., Lee, J., Maddison, C.J., and Dahl, G.E. (2019). On empirical comparisons of optimizers for deep learning. arXiv.
31. Nado, Z., Gilmer, J.M., Shallue, C.J., Anil, R., and Dahl, G.E. (2021). A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. arXiv.
32. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT.
33. Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. (2017). Visualizing the loss landscape of neural nets. arXiv.
34. Bottou, L. (2010, January 22–27). Large-scale machine learning with stochastic gradient descent. Proceedings of the COMPSTAT’2010: 19th International Conference on Computational Statistics, Paris, France.10.1007/978-3-7908-2604-3_16 ↗
35. Adaptive subgradient methods for online learning and stochastic optimization
36. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
37. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
38. Zeiler, M.D. (2012). Adadelta: An adaptive learning rate method. arXiv.
39. Incorporating Nesterov momentum into Adam
40. Asirra: A CAPTCHA that exploits interest-aligned manual image categorization
41. Roy, P., Ghosh, S., Bhattacharya, S., and Pal, U. (2018). Effects of degradations on deep neural network architectures. arXiv.
42. Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv.
43. A survey of the recent architectures of deep convolutional neural networks10.1007/s10462-020-09825-6 ↗
44. Review of deep learning algorithms and architectures10.1109/ACCESS.2019.2912200 ↗
45. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗