Fractional Derivative Gradient-Based Optimizers for Neural Networks and Human Activity Recognition
Source abstract
Abstract
In this paper, fractional calculus principles are considered to implement fractional derivative gradient optimizers for the Tensorflow backend. The performance of these fractional derivative optimizers is compared with that of other well-known ones. Our experiments consider some human activity recognition (HAR) datasets, and the results show that there is a subtle difference between the performance of the proposed method and other existing ones. The main conclusion is that fractional derivative gradient descent optimizers could help to improve the performance of training and validation tasks and opens the possibility to include more fractional calculus concepts to neural networks applied to HAR.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and ELM · Fractional Differential Equations Solutions · Advanced Control Systems Design
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Departamento de Sistemas, Universidad Autónoma Metropolitana, Mexico City 02200, Mexico
References (33)
Show source-provided references
1. Haykin, S.S. (2009). Neural Networks and Learning Machines, Pearson Education. [3rd ed.].
2. Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv.
3. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
4. Zeiler, M.D. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv.
5. Tieleman, T., and Hinton, G. (2012). Neural Networks for Machine Learning, COURSERA. Technical Report.
6. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization, 2014. Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA, USA.
7. Heo, B., Chun, S., Oh, S.J., Han, D., Yun, S., Kim, G., Uh, Y., and Ha, J.W. (2021, January 3–7). AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event.
8. Mathematics in Science and Engineering
9. Oustaloup, A. (1995). La dérivation non Entière: Théorie, Synthèse et Applications, Hermes Science Publications.
10. Luchko, Y. (2021). Fractional Integrals and Derivatives: “True” versus “False”, MDPI.
11. Miller, K., and Ross, B. (1993). An Introduction to the Fractional Calculus and Fractional Differential Equations, Wiley.
12. Fractional-Order Deep Backpropagation Neural Network10.1155/2018/7361628 ↗
13. Fractional-order gradient descent learning of BP neural networks with Caputo derivative10.1016/j.neunet.2017.02.007 ↗
14. Recent history of fractional calculus10.1016/j.cnsns.2010.05.027 ↗
15. Mainardi, F. (2022). Fractional Calculus and Waves in Linear Viscoelasticity, World Scientific. [2nd ed.]. Number 2.
16. Muresan, C.I., Birs, I., Ionescu, C., Dulf, E.H., and De Keyser, R. (2022). A Review of Recent Developments in Autotuning Methods for Fractional-Order Controllers. Fractal Fract., 6.10.3390/fractalfract6010037 ↗
17. The construction of operational matrix of fractional integration for solving fractional differential and integro-differential equations10.1007/s00521-017-3163-9 ↗
18. Gonzalez, E.A., and Petráš, I. (2015, January 27–30). Advances in fractional calculus: Control and signal processing applications. Proceedings of the 2015 16th International Carpathian Control Conference (ICCC), Szilvasvarad, Hungary.10.1109/CarpathianCC.2015.7145064 ↗
19. Henriques, M., Valério, D., Gordo, P., and Melicio, R. (2021). Fractional-Order Colour Image Processing. Mathematics, 9.10.3390/math9050457 ↗
20. Adaptive step-size modified fractional least mean square algorithm for chaotic time series prediction10.1088/1674-1056/23/5/050503 ↗
21. Tarasov, V.E. (2019). On History of Mathematical Economics: Application of Fractional Calculus. Mathematics, 7.10.3390/math7060509 ↗
22. Discrete Fractional-Order BAM Neural Networks with Leakage Delay: Existence and Stability Results10.1002/asjc.1918 ↗
23. Chapter 2—Fractional Derivatives and Integrals
24. Garrappa, R., Kaslik, E., and Popolizio, M. (2019). Evaluation of Fractional Integrals and Derivatives of Elementary Functions: Overview and Tutorial. Mathematics, 7.10.3390/math7050407 ↗
25. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2022, September 08). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: tensorflow.org.
26. Chollet, F., Zhu, Q.S., Rahman, F., Lee, T., Marmiesse, G., Zabluda, O., Qian, C., Jin, H., Watson, M., and Chao, R. (2022, July 04). Keras. Available online: https://keras.io/.
27. Google Inc. (2022, July 04). Google Colab. Available online: https://colab.research.google.com.
28. The mnist database of handwritten digit images for machine learning research10.1109/MSP.2012.2211477 ↗
29. (2022, July 04). Actitracker. Available online: http://www.cis.fordham.edu/wisdm/dataset.php.
30. Reyes-Ortiz, J.L., Anguita, D., Ghio, A., Oneto, L., and Parra, X. (2022, July 04). Human Activity Recognition Using Smartphones Dataset. Available online: https://archive.ics.uci.edu/ml/machine-learning-databases/00240.
31. (2022, July 04). HAR Using CNN in Keras. Available online: https://github.com/Shahnawax/HAR-CNN-Keras.
32. Jason, B. (2022, July 04). How to Model Human Activity from Smartphone Data. Available online: https://machinelearningmastery.com/how-to-model-human-activity-from-smartphone-data/.
33. A Novel Fractional Gradient-Based Learning Algorithm for Recurrent Neural Networks10.1007/s00034-017-0572-z ↗