Motion Planning of Robot Manipulators for a Smoother Path Using a Twin Delayed Deep Deterministic Policy Gradient with Hindsight Experience Replay
Source abstract
Abstract
In order to enhance performance of robot systems in the manufacturing industry, it is essential to develop motion and task planning algorithms. Especially, it is important for the motion plan to be generated automatically in order to deal with various working environments. Although PRM (Probabilistic Roadmap) provides feasible paths when the starting and goal positions of a robot manipulator are given, the path might not be smooth enough, which can lead to inefficient performance of the robot system. This paper proposes a motion planning algorithm for robot manipulators using a twin delayed deep deterministic policy gradient (TD3) which is a reinforcement learning algorithm tailored to MDP with continuous action. Besides, hindsight experience replay (HER) is employed in the TD3 to enhance sample efficiency. Since path planning for a robot manipulator is an MDP (Markov Decision Process) with sparse reward and HER can deal with such a problem, this paper proposes a motion planning algorithm using TD3 with HER. The proposed algorithm is applied to 2-DOF and 3-DOF manipulators and it is shown that the designed paths are smoother and shorter than those designed by PRM.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Robotic Path Planning Algorithms · Autonomous Vehicle Technology and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Information Engineering, Seoul National University of Science and Technology, Seoul 01811, Korea
Robotics R&D Group, Korea Institute of Industrial Technology (KITECH), Ansan 15588, Korea
Funding
Ministry of Trade, Industry and Energy · Korea Institute of Energy Technology Evaluation and Planning
References (36)
Show source-provided references
1. Laumond, J.P. (1998). Robot Motion Planning and Control, Springer.10.1007/BFb0036069 ↗
2. Choset, H.M., Hutchinson, S., Lynch, K.M., Kantor, G., Burgard, W., Kavraki, L.E., and Thrun, S. (2005). Principles of Robot Motion: Theory, Algorithms, and Implementation, MIT Press.
3. Cao, B., Doods, G., and Irwin, G.W. (1994, January 8–13). Time-optimal and smooth constrained path planning for robot manipulators. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, San Diego, CA, USA.
4. Smooth local-path planning for autonomous vehicles110.1177/027836499701600301 ↗
5. Rufli, M., Ferguson, D., and Siegwart, R. (2009, January 12–17). Smooth path planning in constrained environments. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.10.1109/ROBOT.2009.5152506 ↗
6. Sampling-based algorithms for optimal motion planning10.1177/0278364911406761 ↗
7. Robot modeling and control10.1109/MCS.2006.252815 ↗
8. Analysis of probabilistic roadmaps for path planning10.1109/70.660866 ↗
9. Probabilistic roadmaps for path planning in high-dimensional configuration spaces10.1109/70.508439 ↗
10. Kavraki, L.E. (1995). Random Networks in Configuration Space for Fast Path Planning. [Ph.D. Thesis, Stanford University].
11. Randomized query processing in robot path planning10.1006/jcss.1998.1578 ↗
12. A Randomized Approach to Robot Path Planning Based on Lazy Evaluation
13. On the probabilistic foundations of probabilistic roadmap planning10.1177/0278364906067174 ↗
14. Kuffner, J.J., and LaValle, S.M. (2000, January 24–28). RRT-connect: An efficient approach to single-query path planning. Proceedings of the 2000 IEEE International Conference on Robotics and Automation, San Francisco, CA, USA.
15. Lavalle, S., and Kuffner, J. (2000). Rapidly-exploring random trees: Progress and prospects. Algorithmic and Computational Robotics: New Directions, CRC Press.
16. Fast marching tree: A fast marching sampling-based method for optimal motion planning in many dimensions10.1177/0278364915577958 ↗
17. Sutton, R.S., and Barto, A.G. (2011). Reinforcement Learning: An Introduction, MIT Press.
18. Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc.. [1st ed.].10.1002/9780470316887 ↗
19. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
20. Sammut, C., and Webb, G.I. (2017). Efficient Exploration in Reinforcement Learning. Encyclopedia of Machine Learning and Data Mining, Springer US.10.1007/978-1-4899-7687-1 ↗
21. Tokic, M. (2010). Adaptive ε-greedy exploration in reinforcement learning based on value differences. Annual Conference on Artificial Intelligence, Springer.10.1007/978-3-642-16111-7_23 ↗
22. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N.M.O., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
23. Hasselt, H.v., Guez, A., and Silver, D. (2016, January 12–17). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.
24. Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. arXiv.
25. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018, January 2–7). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11796 ↗
26. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 20–22). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
27. Degris, T., Pilarski, P.M., and Sutton, R.S. (2012, January 27–29). Model-free reinforcement learning with continuous action in practice. Proceedings of the 2012 American Control Conference (ACC), Montreal, QC, Canada.10.1109/ACC.2012.6315022 ↗
28. Degris, T., White, M., and Sutton, R.S. (2012). Off-policy actor-critic. arXiv.
29. Bae, H., Kim, G., Kim, J., Qian, D., and Lee, S. (2019). Multi-Robot Path Planning Method Using Reinforcement Learning. Appl. Sci., 9.10.3390/app9153057 ↗
30. Path Planning via an Improved DQN-Based Learning Policy10.1109/ACCESS.2019.2918703 ↗
31. Paul, S., and Vig, L. (2017, January 22–29). Deterministic policy gradient based robotic path planning with continuous action spaces. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCVW.2017.91 ↗
32. Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989385 ↗
33. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, O.P., and Zaremba, W. (2017, January 4–9). Hindsight experience replay. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
34. Cox, I.J., and Wilfong, G.T. (1990). Spatial Planning: A Configuration Space Approach. Autonomous Robot Vehicles, Springer New York.10.1007/978-1-4613-8997-2 ↗
35. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 22–24). Deterministic Policy Gradient Algorithms. Proceedings of the 31st International Conference on International Conference on Machine Learning, Bejing, China.
36. Sutton, R.S., Mcallester, D., Singh, S., and Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 12, MIT Press.