Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
Source abstract
Abstract
Temporal difference (TD) learning is a powerful framework for value function approximation in reinforcement learning. However, standard TD methods often struggle with feature representation and off-policy learning challenges. In this paper, we propose a novel framework, online attentive kernel-based off-policy TD learning, and in combination with well-known algorithms, introduce OAKGTD2, OAKTDC, and OAKETD. This framework uses two-timescale optimization. In the slow-timescale, a sparse representation of state features is learned using an online attentive kernel-based method. In the fast-timescale, auxiliary variables are used to update the value function parameters under the off-policy setting. We theoretically prove the convergence of all three algorithms. Through experiments conducted in several standard reinforcement learning environments, we demonstrate the effectiveness of the improved algorithms and compare their performance with existing algorithms. Specifically, from the perspective of cumulative rewards, the proposed algorithm achieves an average improvement of 15% compared to on-policy algorithms and an average improvement of 25% compared to common off-policy algorithms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
Funding
National Natural Science Foundation of China
References (40)
Show source-provided references
1. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
2. Highly accurate protein structure prediction with AlphaFold10.1038/s41586-021-03819-2 ↗
3. OpenAI (2023). Gpt-4 technical report. arXiv.
4. Champion-level drone racing using deep reinforcement learning10.1038/s41586-023-06419-4 ↗
5. Spatial memory-augmented visual navigation based on hierarchical deep reinforcement learning in unknown environments10.1016/j.knosys.2023.111358 ↗
6. Sutton, R.S. (2018). Reinforcement Learning: An Introduction, MIT Press. A Bradford Book.
7. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
8. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
9. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
10. The utility of sparse representations for control in reinforcement learning
11. Liu, V. (2019). Sparse Representation Neural Networks for Online Reinforcement Learning. [Ph.D. Thesis, University of Alberta].
12. Online attentive kernel-based temporal difference learning10.1016/j.knosys.2023.110902 ↗
13. Sutton, R.S. (1995). Generalization in reinforcement learning: Successful examples using sparse coarse coding. Adv. Neural Inf. Process. Syst., 8.
14. Krawiec, K., and Szubert, M.G. (2011, January 12–16). Learning n-tuple networks for Othello by coevolutionary gradient search. Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, Dublin, Ireland.10.1145/2001576.2001626 ↗
15. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
16. Dropout: A simple way to prevent neural networks from overfitting
17. Makhzani, A., and Frey, B. (2013). K-sparse autoencoders. arXiv.
18. Makhzani, A., and Frey, B.J. (2015). Winner-take-all autoencoders. Adv. Neural Inf. Process. Syst., 28.
19. L 1-regularization path algorithm for generalized linear models10.1111/j.1467-9868.2007.00607.x ↗
20. Regularization theory and neural networks architectures10.1162/neco.1995.7.2.219 ↗
21. Kernel-based reinforcement learning10.1023/A:1017928328829 ↗
22. Practical kernel-based reinforcement learning
23. Micchelli, C.A., Xu, Y., and Zhang, H. (2006). Universal Kernels. J. Mach. Learn. Res., 7.
24. Learning to predict by the methods of temporal differences10.1007/BF00115009 ↗
25. An emphatic approach to the problem of off-policy temporal-difference learning
26. Mou, W., Pananjady, A., and Wainwright, M.J. (2020). Optimal oracle inequalities for solving projected fixed-point equations. arXiv.
27. Online selective kernel-based temporal difference learning10.1109/TNNLS.2013.2270561 ↗
28. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015, January 6–11). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France.
29. Chung, W., Nath, S., Joseph, A., and White, M. (2019, January 6–9). Two-Timescale Networks for Nonlinear Value Function Approximation. Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA.
30. Precup, D., Sutton, R.S., and Dasgupta, S. (July, January 28). Off-policy temporal-difference learning with function approximation. Proceedings of the ICML, Williamstown, MA, USA.
31. Sutton, R.S., Maei, H.R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., and Wiewiora, E. (2009, January 14–18). Fast gradient-descent methods for temporal-difference learning with linear function approximation. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.10.1145/1553374.1553501 ↗
32. Q-learning10.1007/BF00992698 ↗
33. Tsitsiklis, J., and Van Roy, B. (1996). Analysis of temporal-diffference learning with function approximation. Adv. Neural Inf. Process. Syst., 9.
34. A convergent O (n) algorithm for off-policy temporal-difference learning with linear function approximation
35. Borkar, V.S., and Borkar, V.S. (2008). Stochastic Approximation: A Dynamical Systems Viewpoint, Springer.10.1007/978-93-86279-38-5 ↗
36. Kushner, H.J., and Clark, D.S. (2012). Stochastic Approximation Methods for Constrained and Unconstrained Systems, Springer Science & Business Media.
37. Teschl, G. (2021). Ordinary Differential Equations and Dynamical Systems, American Mathematical Society.
38. Stochastic approximation with two time scales10.1016/S0167-6911(97)90015-3 ↗
39. Convergent activation dynamics in continuous time networks10.1016/0893-6080(89)90018-X ↗
40. The ODE method for convergence of stochastic approximation and reinforcement learning10.1137/S0363012997331639 ↗