Extended Maximum Actor–Critic Framework Based on Policy Gradient Reinforcement for System Optimization
Source abstract
Abstract
Recently, significant research efforts have been directed toward leveraging Artificial Intelligence for sensor data processing and system control. In particular, it is essential to determine the optimal path and trajectory by calculating sensor data for effective control systems. For instance, model-predictive control based on Proportional-Integral-Derivative models is intuitive, efficient, and provides outstanding control performance. However, challenges in tracking persist, which requires active research and development to integrate and optimize the control system in terms of Machine Learning. Specifically, Reinforcement Learning, a branch of Machine Learning, has been used in several research fields to solve optimal control problems. In this paper, we propose an Extended Maximum Actor–Critic using a Reinforcement Learning-based method to combine the advantages of both value and policy to enhance the learning stability of actor–critic for optimization of system control. The proposed method integrates the actor and the maximized actor in the learning process to evaluate and identify actions with the highest value, facilitating effective learning exploration. Additionally, to enhance the efficiency and robustness of the agent learning process, we propose Prioritized Hindsight Experience Replay, combining the advantages of Prioritized Experience Replay and Hindsight Experience Replay. To verify this, we performed evaluations and experiments to examine the improved training stability in the MuJoCo environment, which is a simulator based on Reinforcement Learning. The proposed Prioritized Hindsight Experience Replay method significantly enhances the experience to be compared with the standard replay buffer and PER in experimental simulators, such as the simple HalfCheetah-v4 and the complex Ant-v4. Thus, Prioritized Hindsight Experience Replay achieves a higher success rate than PER in FetchReach-v2, demonstrating the significant effectiveness of our proposed method in more complex reward environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Robotic Path Planning Algorithms · Advanced Software Engineering Methodologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic Information System Engineering, Sangmyung University, Cheonan 31066, Republic of Korea
Department of Computer Information and Communication Engineering, Sangmyung University, Cheonan 31066, Republic of Korea
Nuclear System Integrity Sensing and Diagnosis Division, Korea Atomic Energy Research Institute, Daejeon 34057, Republic of Korea
Funding
National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT)
References (38)
Show source-provided references
1. Learning dexterous in-hand manipulation10.1177/0278364919887447 ↗
2. Lobbezoo, A., and Kwon, H.J. (2023). Simulated and Real Robotic Reach, Grasp, and Pick-and-Place Using Combined Reinforcement Learning and Traditional Controls. Robotics, 12.10.3390/robotics12010012 ↗
3. Fayjie, A.R., Hossain, S., Oualid, D., and Lee, D.J. (2018, January 26–30). Driverless Car: Autonomous Driving Using Deep Reinforcement Learning in Urban Environment. Proceedings of the 15th International Conference on Ubiquitous Robots, Honolulu, HI, USA.10.1109/URAI.2018.8441797 ↗
4. Kim, J., Park, S., Kim, J., and Yoo, J. (2023). A Deep Reinforcement Learning Strategy for Surrounding Vehicles-Based Lane-Keeping Control. Sensors, 23.10.3390/s23249843 ↗
5. Grandmaster level in StarCraft II using multi-agent Reinforcement Learning10.1038/s41586-019-1724-z ↗
6. Scheller, C., Schraner, Y., and Vogel, M. (2019, January 8–14). Sample efficient Reinforcement Learning through learning from demonstrations in minecraft. Proceedings of the NeurIPS 2019 Competition and Demonstration Track, Vancouver, CA, Canada. PMLR.
7. A Markovian decision process
8. Konda, V.R., and Tsitsiklis, J.N. (2000, January 20). Actor–critic algorithms. Proceedings of the Advances in Neural Information Processing Systems 12, Chicago, IL, USA.
9. Q-learning10.1007/BF00992698 ↗
10. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., and Wierstra, D. (2015). Continuous control with deep Reinforcement Learning. arXiv.
11. Policy gradient methods for Reinforcement Learning with function approximation
12. An adaptive genetic algorithm for robot motion planning in 2D complex environments10.1016/j.compeleceng.2014.12.014 ↗
13. Adaptive neural network control of an uncertain robot with full-state constraints10.1109/TCYB.2015.2411285 ↗
14. An improved ant colony algorithm for robot path planning10.1007/s00500-016-2161-7 ↗
15. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection10.1177/0278364917710318 ↗
16. Long-range indoor navigation with PRM-RL10.1109/TRO.2020.2975428 ↗
17. Wang, X., Xiong, W., Wang, H., and William, Y.W. (2018, January 8–14). Look before you leap: Bridging model-free and model-based Reinforcement Learning for planned-ahead vision-and-language navigation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01270-0_3 ↗
18. Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., Heinrich, K., Andrew, L., Simon, G., Víctor, V., and Amir Sadik, J.S. (2016). Deepmind lab. arXiv.
19. Loiacono, D., Cardamone, L., and Lanzi, P.L. (2013). Simulated car racing championship: Competition software manual. arXiv.
20. Inamura, T., and Mizuchi, Y. (2020). SIGVerse: A cloud-based VR platform for research on social and embodied human-robot interaction. arXiv.10.3389/frobt.2021.549360 ↗
21. Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., and Zhu, Y. (2013). An interactive 3d environment for visual ai. arXiv.
22. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
23. Mastering the game of Go without human knowledge10.1038/nature24270 ↗
24. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
25. Dynamic programming10.1126/science.153.3731.34 ↗
26. Human-level control through deep Reinforcement Learning10.1038/nature14236 ↗
27. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 6–11). Trust Region Policy Optimization. Proceedings of the International Conference on Machine Learning, Lille, France. PMLR.
28. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization algorithms. arXiv.
29. Fujimoto, S., Hoof, H., and Meger, D. (2018, January 25–31). Addressing function approximation error in actor–critic methods. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden. PMLR.
30. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 25–31). Soft Actor–Critic: Off-Policy maximum entropy deep Reinforcement Learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden. PMLR.
31. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., and Levine, S. (2018). Soft Actor–Critic algorithms and applications. arXiv.
32. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized Experience Replay. arXiv.
33. Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., and Silver, D. (2018). Distributed Prioritized Experience Replay. arXiv.
34. Hindsight Experience Replay
35. Nguyen, H., La, H., and Deans, M. (2019, January 19–22). Hindsight Experience Replay with Experience Ranking. Proceedings of the 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), Oslo, Norway.10.1109/DEVLRN.2019.8850705 ↗
36. Todorov, E., Erez, T., and Tassa, Y.Y. (2012, January 7–12). Mujoco: A physics engine for model-based control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal.10.1109/IROS.2012.6386109 ↗
37. dm_control: Software and tasks for continuous control10.1016/j.simpa.2020.100022 ↗
38. Plappert, M. (2018). Multi-goal Reinforcement Learning: Challenging robotics environments and request for research. arXiv.