Learning an Efficient Gait Cycle of a Biped Robot Based on Reinforcement Learning and Artificial Neural Networks
Source abstract
Abstract
Programming robots for performing different activities requires calculating sequences of values of their joints by taking into account many factors, such as stability and efficiency, at the same time. Particularly for walking, state of the art techniques to approximate these sequences are based on reinforcement learning (RL). In this work we propose a multi-level system, where the same RL method is used first to learn the configuration of robot joints (poses) that allow it to stand with stability, and then in the second level, we find the sequence of poses that let it reach the furthest distance in the shortest time, while avoiding falling down and keeping a straight path. In order to evaluate this, we focus on measuring the time it takes for the robot to travel a certain distance. To our knowledge, this is the first work focusing both on speed and precision of the trajectory at the same time. We implement our model in a simulated environment using q-learning. We compare with the built-in walking modes of an NAO robot by improving normal-speed and enhancing robustness in fast-speed. The proposed model can be extended to other tasks and is independent of a particular robot model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotic Locomotion and Control · Prosthetics and Rehabilitation Robotics · Reinforcement Learning in Robotics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Centro de Investigación en Computación, Instituto Politécnico Nacional, Ciudad de México 07738, Mexico
Tecnológico de Monterrey, Campus Guadalajara, Zapopan 45138, Mexico
References (34)
Show source-provided references
1. Shirashaka, S., Machida, T., and Igarashi, H. (2006, January 18–21). Leg Selectable Interface for Walking Robots on Irregular Terrain. Proceedings of the 2006 SICE-ICASE International Joint Conference, Busan, Korea.10.1109/SICE.2006.314936 ↗
2. Ill-Woo, P., Jung-Yup, K., and Jungho, L. (2006, January 15–19). Online Free Walking Trajectory Generation for Biped. Proceedings of the IEEE Internactional Conference on Robotics and Automation (ICRA 2006), Orlando, FL, USA.
3. Huang, Q., Kajita, S., Koyachi, N., Kaneko, K., Yokoi, K., Kotoku, T., Arai, H., Komoriya, K., and Tanie, K. (1999, January 17–21). Walking patterns and actuator specifications for a biped robot. Proceedings of the 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289), Kyongju, Korea.
4. Tay, A.B. (2009). Walking Nao Omnidirectional Bipedal Locomotion. [Bachelor’s Thesis, Bachelor of Science, School of Computer Science and Engineering, The University of New South Wales].
5. Morimoto, J., Cheng, G., Atkeson, C.G., and Zeglin, G. (May, January 26). A simple reinforcement learning algorithm for biped walking. Proceedings of the IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA ‘04. 2004, New Orleans, LA, USA.10.1109/ROBOT.2004.1307522 ↗
6. Gait Balance and Acceleration of a Biped Robot10.1109/ACCESS.2016.2570255 ↗
7. Michel, O. (1998). Webots: Symbiosis between virtual and real mobile robots. International Conference on Virtual Worlds, Springer. Lecture Notes in Computer Science 1434.10.1007/3-540-68686-X_24 ↗
8. Walking and running with passive compliance: Lessons from engineering a live demonstration of the ATRIAS biped10.1109/MRA.2017.2783922 ↗
9. Martin, W.C., Wu, A., and Geyer, H. (2015, January 26–30). Robust spring mass model running for a physical bipedal robot. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7140085 ↗
10. A New Automatic Gait Cycle Partitioning Method and Its Application to Human Identification10.5391/IJFIS.2017.17.2.51 ↗
11. Walking Control Algorithm of Biped Humanoid Robot on Uneven and Inclined Floor10.1007/s10846-006-9107-8 ↗
12. Planning Walking Patterns for a Biped Robot10.1109/70.938385 ↗
13. Deviation Correction Control of Biped Robot Walking Path Planning10.1088/1757-899X/394/3/032029 ↗
14. Design and adaptive balance control of a biped robot with fewer actuators for slope walking10.1016/j.mechatronics.2017.11.007 ↗
15. Rhythmic-Reflex Hybrid Adaptive Walking Control of Biped Robot
16. Righetti, L., and Jan Ijspeert, A. (2006, January 15–19). Programmable Central Pattern Generators: An application to biped locomotion control. Proceedings of the IEEE International Conference on Robotics and Automation, Orlando, FL, USA.
17. Liu, J., Chen, X., and Veloso, M. (2010). Simplified walking: A new way to generate flexible biped patterns. Mobile Robotics: Solutions and Challenges, World Scientific.10.1142/9789814291279_0071 ↗
18. Rivas, F.M., Canas, J.M., and González, J. (2011, January 28–29). Automatic learning of walking modes for a humanoid robot (in Spanish). Proceedings of the Robot2011 III Robotics Workshop, Experimental robotics, Seville, Spain.
19. Meriçli, Ç., and Veloso, M. (2010, January 9–14). Biped walk learning on NAO through playback and real-time corrective demonstration. Workshop on Agents Learning Interactively from Human Teachers. Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), Toronto, ON, Canada.
20. The Study on the Learning of Walking Gaits for Biped Robots
21. Efficient Walking Gait Generation via Principal Component Representation of Optimal Trajectories: Application to a Planar Biped Robot with Elastic Joints10.1109/LRA.2018.2807578 ↗
22. Corral, E., Marques, F., García, M.J.G., Flores, P., and García-Prada, J.C. (2019). Passive walking biped model with dissipative contact and friction forces. European Conference on Mechanism Science, Springer.10.1007/978-3-319-98020-1_5 ↗
23. Dekker, M. (2009). Zero-Moment Point Method for Stable Biped Walking, University of Technology.
24. Omnidirectional Walking Using ZMP and Preview Control for the NAO Humanoid Robot10.1007/978-3-642-11876-0_33 ↗
25. Lee, H.W., Yang, J.L., Zhang, S.Q., and Chen, Q. (2018, January 23–27). Research on the Stability of Biped Robot Walking on Different Road Surfaces. Proceedings of the 2018 1st IEEE International Conference on Knowledge Innovation and Invention (ICKII), Jeju, Korea.10.1109/ICKII.2018.8569084 ↗
26. Van der Noot, N., and Barrera, A. (2014, January 13–16). Zero-Moment Point on a Bipedal Robot. Proceedings of the 17th IEEE Mediterranean Electrotechnical Conference, Beirut, Lebanon.
27. Liu, Y., Zang, X., Zhang, N., Liu, Y., and Wu, M. (July, January 30). Effects of unilateral restriction of the metatarsophalangeal joints on biped robot walking. Proceedings of the 2018 Eighth International Conference on Information Science and Technology (ICIST), Cordoba, Spain.10.1109/ICIST.2018.8426106 ↗
28. Walking with perturbations: A guide for biped humans and robots10.1088/1748-3190/aada54 ↗
29. Biped Balance Control by Reinforcement Learning
30. McGinnis, P.M. (2015). Biomechanics of Sport and Exercise, Human Kinetics Publishers.
31. Mnih, V., Kavukcuoglu, K., Silver, D., and Graves, A. (arXiv, 2013). Playing Atari with Deep Reinforcement Learning, arXiv.
32. Singla, A., Bhattacharya, S., Dholakiya, D., Bhatnagar, S., Ghosal, A., Amrutur, B., and Kolathaya, S. (arXiv, 2018). Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives, arXiv.10.1109/ICRA.2019.8794179 ↗
33. Rai, A., Antonova, R., Meier, F., and Atkeson, C.G. (arXiv, 2018). Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots, arXiv.
34. Webots: Professional Mobile Robot Simulation10.5772/5618 ↗