Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep-Reinforcement-Learning Approach
Source abstract
Abstract
Industrial robot manipulators are playing a significant role in modern manufacturing industries. Though peg-in-hole assembly is a common industrial task that has been extensively researched, safely solving complex, high-precision assembly in an unstructured environment remains an open problem. Reinforcement-learning (RL) methods have proven to be successful in autonomously solving manipulation tasks. However, RL is still not widely adopted in real robotic systems because working with real hardware entails additional challenges, especially when using position-controlled manipulators. The main contribution of this work is a learning-based method to solve peg-in-hole tasks with hole-position uncertainty. We propose the use of an off-policy, model-free reinforcement-learning method, and we bootstraped the training speed by using several transfer-learning techniques (sim2real) and domain randomization. Our proposed learning framework for position-controlled robots was extensively evaluated in contact-rich insertion tasks in a variety of environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Reinforcement Learning in Robotics · Robotic Path Planning Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Engineering Science, Osaka University, Osaka 560-8531, Japan
Automation Research Team, Industrial CPS Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Tokyo 135-0064, Japan
References (46)
Show source-provided references
1. Kroemer, O., Niekum, S., and Konidaris, G. (2019). A review of robot learning for manipulation: Challenges, representations, and algorithms. arXiv.
2. Quasi-Static Assembly of Compliantly Supported Rigid Parts10.1115/1.3149634 ↗
3. Tsuruoka, T., Fujioka, H., Moriyama, T., and Mayeda, H. (1997, January 7–9). 3D analysis of contact in peg-hole insertion. Proceedings of the 1997 IEEE International Symposium on Assembly and Task Planning (ISATP’97)-Towards Flexible and Agile Assembly and Manufacturing, Marina del Rey, CA, USA.
4. Force control for a rigid dual peg-in-hole assembly10.1108/AA-09-2016-120 ↗
5. Fukumoto, Y., and Harada, K. (2018, January 6–9). Force Control Law Selection for Elastic Part Assembly from Human Data and Parameter Optimization. Proceedings of the 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), Beijing, China.10.1109/HUMANOIDS.2018.8624968 ↗
6. Robot learning of industrial assembly task via human demonstrations10.1007/s10514-018-9725-6 ↗
7. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
8. A learning framework of adaptive manipulative skills from human to robot10.1109/TII.2018.2826064 ↗
9. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection10.1177/0278364917710318 ↗
10. Pinto, L., and Gupta, A. (2016, January 16–21). Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487517 ↗
11. Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. Proceedings of the 2017 IEEE international conference on robotics and automation (ICRA), Singapore.10.1109/ICRA.2017.7989385 ↗
12. Learning the peg-into-hole assembly operation with a connectionist reinforcement technique10.1016/S0166-3615(97)00015-8 ↗
13. Inoue, T., De Magistris, G., Munawar, A., Yokoya, T., and Tachibana, R. (2017, January 24–28). Deep reinforcement learning for high precision assembly tasks. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202244 ↗
14. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
15. Feedback deep deterministic policy gradient with fuzzy reward for robotic multiple peg-in-hole assembly tasks10.1109/TII.2018.2868859 ↗
16. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M.A. (2014, January 22–24). Deterministic Policy Gradient Algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China.
17. Fan, Y., Luo, J., and Tomizuka, M. (2019, January 20–24). A learning framework for high precision industrial assembly. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793659 ↗
18. Levine, S., and Koltun, V. (2013, January 16–21). Guided policy search. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA.
19. Luo, J., Solowjow, E., Wen, C., Ojea, J.A., and Agogino, A.M. (2018, January 1–5). Deep reinforcement learning for robotic assembly of mixed deformable and rigid objects. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594353 ↗
20. Learning-based variable compliance control for robotic assembly10.1115/1.4041331 ↗
21. Learning variable impedance control10.1177/0278364911402527 ↗
22. Lee, M.A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., and Bohg, J. (2019, January 20–24). Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793485 ↗
23. Adaptation of manipulation skills in physical contact with the environment to reference force profiles10.1007/s10514-015-9435-2 ↗
24. Learning Force Control for Contact-rich Manipulation Tasks with Rigid Position-controlled Robots10.1109/LRA.2020.3010739 ↗
25. Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., and Fox, D. (2019, January 20–24). Closing the sim-to-real loop: Adapting simulation randomization with real world experience. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793789 ↗
26. Learning dexterous in-hand manipulation10.1177/0278364919887447 ↗
27. Sharma, K., Shirwalkar, V., and Pal, P.K. (2013, January 16–18). Intelligent and environment-independent peg-in-hole search strategies. Proceedings of the 2013 International Conference on Control, Automation, Robotics and EMbedded Systems (CARE), Jabalpur, India.10.1109/CARE.2013.6733716 ↗
28. Zakharov, S., Shugurov, I., and Ilic, S. (November, January 27). Dpod: 6d pose object detector and refiner. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
29. Peng, S., Liu, Y., Huang, Q., Zhou, X., and Bao, H. (2019, January 16–20). Pvnet: Pixel-wise voting network for 6dof pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00469 ↗
30. Xiang, Y., Schmidt, T., Narayanan, V., and Fox, D. (2018). PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Robot. Sci. Syst. (RSS), 2018.10.15607/RSS.2018.XIV.019 ↗
31. Hodan, T., Haluza, P., Obdržálek, Š., Matas, J., Lourakis, M., and Zabulis, X. (2017, January 24–31). T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.103 ↗
32. Harada, K., Nakayama, K., Wan, W., Nagata, K., Yamanobe, N., and Ramirez-Alpizar, I.G. (2018). Tool exchangeable grasp/assembly planner. Proceedings of the International Conference on Intelligent Autonomous Systems, Springer.10.1007/978-3-030-01370-7_62 ↗
33. ASPPR: A new Assembly Sequence and Path Planner/Replanner for monotone and nonmonotone assembly planning10.1016/j.cad.2020.102828 ↗
34. Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018, January 21–25). Overcoming exploration in reinforcement learning with demonstrations. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.10.1109/ICRA.2018.8463162 ↗
35. Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. (November, January 30). Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning. Proceedings of the Conference on Robot Learning (CoRL) 2019, Osaka, Japan.
36. Wang, Y., Harada, K., and Wan, W. (2020). Motion planning of skillful motions in assembly process through human demonstration. Adv. Robot., 1–15.10.1080/01691864.2020.1782260 ↗
37. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv.
38. Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. (May, January 30). Distributed Prioritized Experience Replay. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
39. Bai, S., Kolter, J.Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv.
40. The parallel approach to force/position control of robotic manipulators10.1109/70.246048 ↗
41. Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J.A., Solowjow, E., and Levine, S. (2019, January 20–24). Residual Reinforcement Learning for Robot Control. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794127 ↗
42. Silver, T., Allen, K.R., Tenenbaum, J.B., and Kaelbling, L.P. (2018). Residual Policy Learning. arXiv.
43. Bellegarda, G., and Byl, K. (2019, January 3–8). Training in Task Space to Speed Up and Guide Reinforcement Learning. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8967995 ↗
44. Koenig, N., and Howard, A. (October, January 28). Design and use paradigms for gazebo, an open-source multi-robot simulator. Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sendai, Japan.
45. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017, January 24–28). Domain randomization for transferring deep neural networks from simulation to the real world. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202133 ↗
46. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12–17). ROS: An open-source Robot Operating System. Proceedings of the ICRA workshop on Open Source Software, Kobe, Japan.