An Effective Optimization Method for Machine Learning Based on ADAM
Source abstract
Abstract
A machine is taught by finding the minimum value of the cost function which is induced by learning data. Unfortunately, as the amount of learning increases, the non-liner activation function in the artificial neural network (ANN), the complexity of the artificial intelligence structures, and the cost function’s non-convex complexity all increase. We know that a non-convex function has local minimums, and that the first derivative of the cost function is zero at a local minimum. Therefore, the methods based on a gradient descent optimization do not undergo further change when they fall to a local minimum because they are based on the first derivative of the cost function. This paper introduces a novel optimization method to make machine learning more efficient. In other words, we construct an effective optimization method for non-convex cost function. The proposed method solves the problem of falling into a local minimum by adding the cost function in the parameter update rule of the ADAM method. We prove the convergence of the sequences generated from the proposed method and the superiority of the proposed method by numerical comparison with gradient descent (GD, ADAM, and AdaMax).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face and Expression Recognition · Stochastic Gradient Optimization Techniques · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Division of Creative Integrated General Studies, Daegu University College, Kyungsan 38453, Korea
Department of Mathematics, College of Natural Sciences, Chungnam National University, Daejeon 34134, Korea
Funding
National Research Foundation of Korea
References (25)
Show source-provided references
1. Deng, L., Li, J., Huang, J., Yao, K., Yu, D., Seide, F., Seltzer, M.L., Zweig, G., He, X., and Williams, J. (2013, January 26–31). Recent advances in deep learning for speech research at microsoft. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing—ICASSP 2013, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639345 ↗
2. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv.10.1007/978-3-642-24797-2_3 ↗
3. Graves, A., Mohamed, A., and Hinton, G. (2013, January 26–31). Speech recognition with deep recurrent neural networks. Proceedings of the 2013 International Conference on Acoustics, Speech and Signal Processing—ICASSP 2013, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638947 ↗
4. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv.
5. Tieleman, T., and Hinton, G.E. (2012). Lecture 6.5—RMSProp, COURSERA: Neural Networks for Machine Learning, COURSERA. Technical Report.
6. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q., Mao, M., Ranzato, M., Senior, A., and Tucker, P. (2012, January 3). Large scale distributed deep networks. Proceedings of the 25th International Conference on Neural Information Processing Systems—NIPS 2012, Lake Tahoe, NV, USA.
7. Jaitly, N., Nguyen, P., Senior, A., and Vanhoucke, V. (2012, January 9–13). Application of pretrained deep neural networks to large vocabulary speech recognition. Proceedings of the INTERSPEECH 2012, 13th Annual Conference of the International Speech Communication Association—ISCA 2012, Portland, OR, USA.10.21437/Interspeech.2012-10 ↗
8. Natural gradient works efficiently in learning10.1162/089976698300017746 ↗
9. Adaptive subgradient methods for online learning and stochastic optimization
10. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups10.1109/MSP.2012.2205597 ↗
11. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3). Imagenet classification with deep convolutional neural networks. Proceedings of the Neural Information Processing Systems–NIPS 2012, Lake Tahoe, NV, USA.
12. Pascanu, R., and Bengio, Y. (2013). Revisiting natural gradient for deep networks. arXiv.
13. Sutskever, I., Martens, J., Dahl, G., and Hinton, G.E. (2013, January 16–21). On the importance of initialization and momentum in deep learning. Proceedings of the 30th International Conference on Machine Learning—ICML 2013, Atlanta, GA, USA. PMLR 28(3).
14. Kochenderfer, M., and Wheeler, T. (2019). Algorithms for Optimization, The MIT Press Cambridge.
15. Kingma, D.P., and Ba, J. (2015, January 7–9). ADAM: A method for stochastic optimization. Proceedings of the 3rd International Conference for Learning Representations—ICLR 2015, San Diego, CA, USA.
16. Optimization Methods for Large-Scale Machine Learning10.1137/16M1080173 ↗
17. Roux, N.L., and Fitzgibbon, A.W. (2010, January 21–24). A fast natural newton method. Proceedings of the 27th International Conference on Machine Learning—ICML 2010, Haifa, Israel.
18. Sohl-Dickstein, J., Poole, B., and Ganguli, S. (2014, January 21–24). Fast large-scale optimization by unifying stochastic gradient and quasi-newton methods. Proceedings of the 31st International Conference on Machine Learning—ICML 2014, Beijing, China.
19. Zeiler, M.D. (2012). Adadelta: An adaptive learning rate method. arXiv.
20. Learning representations by back-propagating errors10.1038/323533a0 ↗
21. Becker, S., and LeCun, Y. (1988). Improving the Convergence of Back-Propagation Learning with Second Order Methods, Department of Computer Science, University of Toronto. Technical Report.
22. Kelley, C.T. (1995). Iterative methods for linear and nonlinear equations. Frontiers in Applied Mathematics, SIAM.10.1137/1.9781611970944 ↗
23. Reddi, S.J., Kale, S., and Kumar, S. (2019). On the Convergence of ADAM and Beyond. arXiv.
24. Ruppert, D. (1988). Efficient Estimations from a Slowly Convergent Robbins-Monro Process, Cornell University Operations Research and Industrial Engineering. Technical Report.
25. Zinkevich, M. (2003, January 21–24). Online Convex Programming and Generalized Infinitesimal Gradient Ascent. Proceedings of the Twentieth International Conference on Machine Learning—ICML-2003, Washington, DC, USA.