Sim–Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning
Source abstract
Abstract
Models trained with Deep Reinforcement learning (DRL) have been deployed in various areas of robotics with varying degree of success. To overcome the limitations of data gathering in the real world, DRL training utilizes simulated environments and transfers the learned policy to real-world scenarios, i.e., sim–real transfer. Simulators fail to accurately capture the entire dynamics of the real world, so simulation-trained policies often fail when applied to reality, termed a reality gap (RG). In this paper, we propose a search (mapping) algorithm that takes in real-world observation (images) and maps them to the policy-equivalent images in the simulated environment using a convolution neural network (CNN) model. The two-step training process, DRL policy and a mapping model, overcomes the RG problem with simulated data only. We evaluated the proposed system with a gripping task of a custom-made robot arm in the real world and compared the performance against a conventional DRL sim–real transfer system. The conventional system achieved a 15–57% success rate in gripping operation depending on the position of the target object while the mapping-based sim–real system achieved 100%. The experimental results demonstrated that the proposed DRL with mapping method appropriately corresponded the real world to the simulated environment, confirming that the scheme can achieve high sim–real generalization at significantly low training costs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Robot Manipulation and Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Intelligent Production Technology Research & Development Center for Aerospace (IPTeCA), Tokai National Higher Education and Research System, Gifu 501-1193, Japan
Center for Robotics and Biomedical Engineering, Dedan Kimathi University of Technology, Nyeri 657-10100, Kenya
Graduate School of Engineering, Gifu University, Gifu 501-1193, Japan
References (29)
Show source-provided references
1. Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., and Dresp, B. (2021). Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review. Robotics, 10.10.3390/robotics10010022 ↗
2. Zhao, W., Queralta, J.P., and Westerlund, T. (2020, January 1–4). Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey. Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, ACT, Australia.10.1109/SSCI47803.2020.9308468 ↗
3. Munikoti, S., Agarwal, D., Das, L., Halappanavar, M., and Natarajan, B. (2022). Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications. arXiv.10.1109/TNNLS.2023.3283523 ↗
4. Dulac-Arnold, G., Mankowitz, D.J., and Hester, T. (2019). Challenges of Real-World Reinforcement Learning. arXiv.
5. Breyer, M., Furrer, F., Novkovic, T., Siegwart, R.Y., and Nieto, J.I. (2018). Flexible Robotic Grasping with Sim-to-Real Transfer based Reinforcement Learning. arXiv.
6. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning10.1109/ACCESS.2021.3126658 ↗
7. Nagabandi, A., Clavera, I., Liu, S., Fearing, R.S., Abbeel, P., Levine, S., and Finn, C. (2019). Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning. arXiv.
8. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017, January 24–28). Domain randomization for transferring deep neural networks from simulation to the real world. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202133 ↗
9. Volpi, R., Larlus, D., and Rogez, G. (2021, January 20–25). Continual Adaptation of Visual Representations via Domain Randomization and Meta-learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00442 ↗
10. Jiang, Y., Zhang, T., Ho, D., Bai, Y., Liu, C.K., Levine, S., and Tan, J. (June, January 30). SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561731 ↗
11. Ho, D., Rao, K., Xu, Z., Jang, E., Khansari, M., and Bai, Y. (June, January 30). RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561157 ↗
12. Arndt, K., Hazara, M., Ghadirzadeh, A., and Kyrki, V. (August, January 31). Meta Reinforcement Learning for Sim-to-real Domain Adaptation. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196540 ↗
13. Traoré, K.R., Caselles-Dupré, H., Lesort, T., Sun, T., Rodríguez, N.D., and Filliat, D. (2019). Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer. arXiv.
14. Zhang, F., Leitner, J., Milford, M., Upcroft, B., and Corke, P. (2015). Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control. arXiv.
15. James, S., and Johns, E. (2016). 3D Simulation for Robot Arm Control with Deep Q-Learning. arXiv.
16. Liu, N., Cai, Y., Lu, T., Wang, R., and Wang, S. (2020). Real–Sim–Real Transfer for Real-World Robot Control Policy Learning with Deep Reinforcement Learning. Appl. Sci., 10.10.3390/app10051555 ↗
17. Rao, K., Harris, C., Irpan, A., Levine, S., Ibarz, J., and Khansari, M. (2020, January 13–19). RL-CycleGan: Reinforcement learning aware simulation-to-real. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01117 ↗
18. Du, Y., Watkins, O., Darrell, T., Abbeel, P., and Pathak, D. (June, January 30). Auto-Tuned Sim-to-Real Transfer. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9562091 ↗
19. Obando-Ceron, J.S., and Castro, P.S. (2021, January 18–24). Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research. Proceedings of the International Conference on Machine Learning, Virtual.
20. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv.
21. Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12–17). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10295 ↗
22. Behzadan, V., and Munir, A. (2017). Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks. arXiv.10.1007/978-3-319-62416-7_19 ↗
23. Spears, T., Jacques, B., Howard, M., and Sederberg, P. (2017). Scale-invariant temporal history (SITH): Optimal slicing of the past in an uncertain world. arXiv.
24. A real-time detection method of building energy efficiency based on image processing10.1016/j.jvcir.2019.02.032 ↗
25. Bui, H.M., Lech, M., Cheng, E., Neville, K., and Burnett, I.S. (2016, January 27–29). Using grayscale images for object recognition with convolutional-recursive neural network. Proceedings of the 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE), Ha-Long, Vietnam.10.1109/CCE.2016.7562656 ↗
26. Zhao, W., Queralta, J.P., Qingqing, L., and Westerlund, T. (2020, January 20–22). Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning. Proceedings of the 2020 5th International Conference on Robotics and Automation Engineering (ICRAE), Singapore.10.1109/ICRAE50850.2020.9310796 ↗
27. Ship detection from coastal surveillance videos via an ensemble Canny-Gaussian-morphology framework10.1017/S0373463321000540 ↗
28. DROID: Minimizing the Reality Gap Using Single-Shot Human Demonstration10.1109/LRA.2021.3062311 ↗
29. Bridging the Model-Reality Gap With Lipschitz Network Adaptation10.1109/LRA.2021.3131698 ↗