Compound Heuristic Information Guided Policy Improvement for Robot Motor Skill Acquisition
Source abstract
Abstract
Discovering the implicit pattern and using it as heuristic information to guide the policy search is one of the core factors to speed up the procedure of robot motor skill acquisition. This paper proposes a compound heuristic information guided reinforcement learning algorithm PI2-CMA-KCCA for policy improvement. Its structure and workflow are similar to a double closed-loop control system. The outer loop realized by Kernel Canonical Correlation Analysis (KCCA) infers the implicit nonlinear heuristic information between the joints of the robot. In addition, the inner loop operated by Covariance Matrix Adaptation (CMA) discovers the hidden linear correlations between the basis functions within the joint of the robot. These patterns which are good for learning the new task can automatically determine the mean and variance of the exploring perturbation for Path Integral Policy Improvement (PI2). Compared with classical PI2, PI2-CMA, and PI2-KCCA, PI2-CMA-KCCA can not only endow the robot with the ability to realize transfer learning of trajectory planning from the demonstration to the new task, but also complete it more efficiently. The classical via-point experiments based on SCARA and Swayer robots have validated that the proposed method has fast learning convergence and can find a solution for the new task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Reinforcement Learning in Robotics · Muscle activation and electromyography studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation, Wuhan University of Technology, Wuhan 430070, China
School of Electronic and Information Engineering, University of Science and Technology Liaoning, Anshan 114051, China
Funding
National Natural Science Foundation of China · Excellent Dissertation Cultivation Funds of Wuhan University of Technology
References (14)
Show source-provided references
1. Siciliano, B., and Khatib, O. (2016). Springer Handbook of Robotics, Springer.10.1007/978-3-319-32552-1 ↗
2. Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions10.1177/0278364915587923 ↗
3. Paraschos, A., Daniel, C., Peters, J., and Neumann, G. (2013, January 5–10). Probabilistic movement primitives. Proceedings of the 27th Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
4. Khansari-Zadeh, S.M., and Billard, A. (2010, January 3–7). BM: An iterative algorithm to learn stable nonlinear dynamical systems with Gaussian mixture models. Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA.10.1109/ROBOT.2010.5510001 ↗
5. Ratliff, N.D., Bagnell, J.A., and Zinkevich, M.A. (2006, January 25–29). Maximum margin planning. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143936 ↗
6. Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning, The MIT Press. A Bradford Book.
7. Natural Actor-Critic10.1016/j.neucom.2007.11.026 ↗
8. Kober, J., and Peters, J. (2009, January 12–17). Learning motor primitives for robotics. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.10.1109/ROBOT.2009.5152577 ↗
9. Peters, J., and Altun, Y. (2010, January 11–15). Relative entropy policy search. Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, Atlanta, GA, USA.
10. A Generalized Path Integral Control Approach to Reinforcement Learning
11. Stulp, F., and Sigaud, O. (July, January 26). Path integral policy improvement with covariance matrix adaptation. Proceedings of the 29 th International Conference on Machine Learning, Edinburgh, UK.
12. Intelligent trajectory planning based on reinforcement learning with KCCA inference for robot
13. Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors10.1162/NECO_a_00393 ↗
14. Appearance models based on kernel canonical correlation analysis10.1016/S0031-3203(03)00058-X ↗