Solving a Simple Geduldspiele Cube with a Robotic Gripper via Sim-to-Real Transfer
Source abstract
Abstract
Geduldspiele cubes (also known as patience cubes in English) are interesting problems to solve with robotic systems on the basis of machine learning approaches. Generally, highly dexterous hand and finger movement is required to solve them. In this paper, we propose a reinforcement-learning-based approach to solve simple geduldspiele cubes of a flat plane, a convex plane, and a concave plane. The key idea of the proposed approach is that we adopt a sim-to-real framework in which a robotic agent is virtually trained in simulation environment based on reinforcement learning, then the virtually trained robotic agent is deployed into a physical robotic system and evaluated for tasks in the real world. We developed a test bed which consists of a dual-arm robot with a patience cube in a gripper and the virtual avatar system to be trained in the simulation world. The experimental results showed that the virtually trained robotic agent was able to solve simple patience cubes in the real world as well. Based on the results, we could expect to solve the more complex patience cubes by augmenting the proposed approach with versatile reinforcement learning algorithms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Reinforcement Learning in Robotics · Robotic Path Planning Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Yonsei University, Wonju 26493, Korea
Division of Software, Yonsei University, Wonju 26493, Korea
Funding
National Research Foundation of Korea (NRF) · Institute of Information and Communications Technology Planning and Evaluation (IITP)
References (47)
Show source-provided references
1. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
2. Zhao, W., Queralta, J.P., and Westerlund, T. (2020, January 1–4). Sim-to-real transfer in deep reinforcement learning for robotics: A survey. Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia.10.1109/SSCI47803.2020.9308468 ↗
3. An introduction to deep reinforcement learning10.1561/2200000071 ↗
4. Li, Y. (2017). Deep reinforcement learning: An overview. arXiv.
5. Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv.
6. A comprehensive survey on safe reinforcement learning
7. Du, Y., Watkins, O., Darrell, T., Abbeel, P., and Pathak, D. (June, January 30). Auto-tuned sim-to-real transfer. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9562091 ↗
8. Jin, S., Zhu, X., Wang, C., and Tomizuka, M. (2021, January 25–28). Contact pose identification for peg-in-hole assembly under uncertainties. Proceedings of the 2021 American Control Conference (ACC), Online.10.23919/ACC50511.2021.9482981 ↗
9. Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., and Fox, D. (2019, January 20–24). Closing the sim-to-real loop: Adapting simulation randomization with real world experience. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793789 ↗
10. Matas, J., James, S., and Davison, A.J. (2018, January 29–31). Sim-to-real reinforcement learning for deformable object manipulation. Proceedings of the Conference on Robot Learning, PMLR, Zürich, Switzerland.
11. Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2017). Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv.10.15607/RSS.2018.XIV.049 ↗
12. Sadeghi, F., and Levine, S. (2016). CAD2RL: Real Single-Image Flight without a Single Real Image. arXiv.10.15607/RSS.2017.XIII.034 ↗
13. Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., and Farhadi, A. (June, January 29). Target-driven visual navigation in indoor scenes using deep reinforcement learning. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989381 ↗
14. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017, January 24–28). Domain randomization for transferring deep neural networks from simulation to the real world. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan.10.1109/IROS.2017.8202133 ↗
15. Bateux, Q., Marchand, E., Leitner, J., Chaumette, F., and Corke, P. (2018, January 21–25). Training deep neural networks for visual servoing. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461068 ↗
16. Arndt, K., Hazara, M., Ghadirzadeh, A., and Kyrki, V. (August, January 31). Meta reinforcement learning for sim-to-real domain adaptation. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Online.10.1109/ICRA40945.2020.9196540 ↗
17. Puang, E.Y., Tee, K.P., and Jing, W. (2020, January 25–29). Kovis: Keypoint-based visual servoing with zero-shot sim-to-real transfer for robotics manipulation. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341370 ↗
18. Todorov, E., Erez, T., and Tassa, Y. (2012, January 7–12). Mujoco: A physics engine for model-based control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.10.1109/IROS.2012.6386109 ↗
19. Per-contact iteration method for solving contact dynamics10.1109/LRA.2018.2792536 ↗
20. Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion10.1109/LRA.2022.3151396 ↗
21. Virtual commissioning of a mobile ore chute
22. Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2018). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. Field and Service Robotics, Springer.10.1007/978-3-319-67361-5_40 ↗
23. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, January 13–15). CARLA: An open urban driving simulator. Proceedings of the Conference on Robot Learning, PMLR, Mountain View, CA, USA.
24. Furrer, F., Burri, M., Achtelik, M., and Siegwart, R. (2016). Rotors—A Modular Gazebo Mav Simulator Framework. Robot Operating System (ROS), Springer.10.1007/978-3-319-26054-9_23 ↗
25. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
26. Kaplanis, C., Shanahan, M., and Clopath, C. (2018, January 10–15). Continual reinforcement learning with complex synapses. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
27. Traoré, R., Caselles-Dupré, H., Lesort, T., Sun, T., Díaz-Rodríguez, N., and Filliat, D. (2019). Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. arXiv.
28. Decentralized control for interconnected uncertain systems: Extensions to higher-order uncertainties10.1080/00207179308934457 ↗
29. Amor, R.B., and Elloumi, S. (2017, January 14–17). On decentralized control techniques of interconnected systems-application to a double-parallel inverted pendulum. Proceedings of the 2017 International Conference on Advanced Systems and Electric Technologies (IC_ASET), Hammamet, Tunisia.
30. Tsividis, P.A., Pouncy, T., Xu, J.L., Tenenbaum, J.B., and Gershman, S.J. (2017, January 27–29). Human learning in Atari. Proceedings of the 2017 AAAI Spring Symposium Series, Stanford, CA, USA.
31. Zhu, H., Gupta, A., Rajeswaran, A., Levine, S., and Kumar, V. (2019, January 20–24). Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montral, QC, Canada.10.1109/ICRA.2019.8794102 ↗
32. Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, S. (2017, January 13–15). One-shot visual imitation learning via meta-learning. Proceedings of the Conference on Robot Learning, PMLR, Mountain View, CA, USA.
33. Yu, T., Abbeel, P., Levine, S., and Finn, C. (2018). One-shot hierarchical imitation learning of compound visuomotor tasks. arXiv.
34. Ng, A.Y., and Russell, S. (July, January 29). Algorithms for inverse reinforcement learning. Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford, CA, USA.
35. Abbeel, P., and Ng, A.Y. (2004, January 4–8). Apprenticeship learning via inverse reinforcement learning. Proceedings of the Twenty-First International Conference on Machine Learning, Banff, AB, Canada.10.1145/1015330.1015430 ↗
36. A new human-based metaheuristic algorithm for solving optimization problems on the base of simulation of driving training process
37. Learning one-shot imitation from humans without humans10.1109/LRA.2020.2977835 ↗
38. Rivera, C.G., Handelman, D.A., Ratto, C.R., Patrone, D., and Paulhamus, B.L. (2022, January 19–24). Visual Goal-Directed Meta-Imitation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00421 ↗
39. Reinforcement learning with replacing eligibility traces10.1007/BF00114726 ↗
40. Hasp Frank, A., and Tjernström, M. (2019). Construction and Theoretical Study of a Ball Balancing Platform: Limitations When Stabilizing Dynamic Systems Through Implementation of Automatic Control Theory, KTH Royal Institute of Technology.
41. Keshmiri, M., Jahromi, A.F., Mohebbi, A., Hadi Amoozgar, M., and Xie, W.F. (2012). Modeling and control of ball and beam system using model based and non-model based control approaches. Int. J. Smart Sens. Intell. Syst., 5.10.21307/ijssis-2017-468 ↗
42. Putting: How a golf ball and hole interact10.1119/1.16592 ↗
43. Dynamics of golf ball-hole interactions: Rolling around the rim
44. Interaction of a golf ball with the flagstick and hole10.1007/s12283-021-00347-0 ↗
45. (2022, September 23). Coefficient of Friction, Rolling Resistance and Aerodynamics, Coefficient of Friction for a Range of Material Combinations. Available online: https://www.tribology-abc.com/abc/cof.htm.
46. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
47. Unity Technologies (2022, September 23). PPO-Specific Configurations. Available online: https://github.com/Unity-Technologies/ml-agents/blob/release_18_docs/docs/Training-Configuration-File.md#ppo-specific-configurations.