Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera
Source abstract
Abstract
End-to-end robot learning has achieved a great success for robots to obtain various manipulation skills. It learns a function which maps visual information to robotic action directly. Because of the diversity of target objects, most end-to-end robot learning approaches have focused on a single object-specific task with a limited capability of generalization. In this work, an object detection-based one-shot learning method is proposed, which separates the semantic understanding from robot control. It enables a robot to acquire similar manipulation skills efficiently and to have the ability to cope with new objects with a single demonstration. This approach mainly has two modules: the object detection network and the motion policy network. With RGB images, the object detection network tries to output the task-related semantic keypoint of the target object, which is the center of the container in this application, and the motion policy network generates the motion action based on the depth map and the detected keypoint. To evaluate this proposed pipeline, a series of experiments are conducted on typical placing tasks in different simulation scenarios and, additionally, the learned policy is transferred from simulation to the real world without any fine-tuning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Human Pose and Action Recognition · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering, Shanghai Jiao Tong University (SJTU), Shanghai 200240, China
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University (SJTU), Shanghai 200240, China
Funding
National Science Foundation of China
References (35)
Show source-provided references
1. End-to-end training of deep visuomotor policies
2. Coline, D., Abbeel, P., Darrell, T., and Levine, S. (2018, January 21–25). Deep object-centric representations for generalizable robot learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.
3. Stephen, J., Davison, A.J., and Johns, E. (2017, January 13–15). Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA.
4. Chelsea, F., Yu, T., Zhang, T., Abbeel, P., and Levine, S. (2017, January 13–15). One-Shot Visual Imitation Learning via Meta-Learning. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA.
5. James, S., Bloesch, M., and Davison, A.J. (2018, January 29–31). Task-Embedded Control Networks for Few-Shot Imitation Learning. Proceedings of the Conference on Robot Learning, Zurich, Switzerland.
6. Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., and Yan, J. (2019, January 16–20). Siamrpn++: Evolution of siamese visual tracking with very deep networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00441 ↗
7. Manuelli, L., Gao, W., Florence, P., and Tedrake, R. (2019). kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation. arXiv.
8. Du, G., Wang, K., and Lian, S. (2019). Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review. arXiv.10.1007/s10462-020-09888-5 ↗
9. Pinto, L., and Gupta, A. (2016, January 16–21). Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487517 ↗
10. Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. (2018, January 1–5). Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593986 ↗
11. Transfer of Robot Perception Module with Adversarial Learning10.1109/ACCESS.2019.2923541 ↗
12. Lampe, T., and Riedmiller, M. (2013, January 4–9). Acquiring visual servoing reaching and grasping skills using neural reinforcement learning. Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN), Dallas, TX, USA.10.1109/IJCNN.2013.6707053 ↗
13. Sascha, L., Riedmiller, M., and Voigtländer, A. (2012, January 10–15). Autonomous reinforcement learning on raw visual input data in a real world application. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN), Brisbane, Australia.
14. Sergey, L., Wagener, N., and Abbeel, P. (2015, January 26–30). Learning contact-rich manipulation skills with guided policy search. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.
15. Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation10.2991/ijcis.d.191017.001 ↗
16. Lee, M.A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., and Bohg, J. (2019, January 20–24). Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793485 ↗
17. Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., and Mrinal Kalakrishnan, V.V. (2018, January 29–31). Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Proceedings of the Conference on Robot Learning, Zurich, Switzerland.
18. Chen, X., Ghadirzadeh, A., Folkesson, J., and Jensfeltc, P. (2018, January 1–5). Deep reinforcement learning to acquire navigation skills for wheel-legged robots in complex environments. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593702 ↗
19. Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P. (2018, January 21–25). Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461249 ↗
20. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2016, January 5–10). Cooperative inverse reinforcement learning. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
21. Rahmatizadeh, R., Abolghasemi, P., Bölöni, L., and Levine, S. (2018, January 21–25). Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461076 ↗
22. Abolghasemi, P., Mazaheri, A., Shah, M., and Bölöni, L. (2019, January 16–20). Pay Attention—Robustifying a Deep Visuomotor Policy Through Task-Focused Visual Attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00438 ↗
23. Hämäläinen, A., Arndt, K., Ghadirzadeh, A., and Kyrki, V. (2019). Affordance Learning for End-to-End Visuomotor Robot Control. arXiv.10.1109/IROS40897.2019.8968596 ↗
24. Chen, X., Ghadirzadeh, A., Björkman, M., and Jensfelt, P. (2019). Adversarial Feature Training for Generalizable Robotic Visuomotor Control. arXiv.10.1109/ICRA40945.2020.9197505 ↗
25. Wang, C., Martín-Martín, R., Xu, D., Lv, J., Lu, C., Fei-Fei, L., Savarese, S., and Zhu, Y. (2019). 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints. arXiv.10.1109/ICRA40945.2020.9196679 ↗
26. Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., and Torr, P.H.S. (2016, January 8–16). Fully-convolutional siamese networks for object tracking. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-48881-3_56 ↗
27. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
28. Li, B., Yan, J., Wu, W., Zhu, Z., and Hu, X. (2018, January 18–23). High performance visual tracking with siamese region proposal network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00935 ↗
29. Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. (2017). One-shot learning for semantic segmentation. arXiv.10.5244/C.31.167 ↗
30. Rakelly, K., Shelhamer, E., Darrell, T., Efros, A.A., and Levine, S. (2018). Few-shot segmentation propagation with guided networks. arXiv.
31. Tai, L., Zhang, J., Liu, M., and Burgard, W. (2018, January 21–25). Socially compliant navigation through raw depth inputs with generative adversarial imitation learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460968 ↗
32. Chen, G., Yu, H., Dong, W., Sheng, X., Zhu, X., and Ding, H. (2019). Learning to Navigate from Simulation via Spatial and Semantic Information Synthesis. arXiv.
33. Morrison, D., Corke, P., and Leitner, J. (2019). Learning robust, real-time, reactive robotic grasping. Int. J. Robot. Res.10.1177/0278364919859066 ↗
34. Qin, Y., Chen, R., Zhu, H., Song, M., Xu, J., and Su, H. (2019). S4G: Amodal Single-view Single-Shot SE (3) Grasp Detection in Cluttered Scenes. arXiv.
35. David, H., and Eck, D. (2017). A neural representation of sketch drawings. arXiv.