Rapid-Learning Collaborative Pushing and Grasping via Deep Reinforcement Learning and Image Masking
Source abstract
Abstract
When multiple objects are positioned close together or stacked, pre-grasp operations such as pushing objects can be used to create space for the grasp, thereby improving the grasping success rate. This study develops a model based on a deep Q-learning network architecture and introduces a fully convolutional network to accurately identify pixels in the workspace image that correspond to target locations for exploration. In addition, this study incorporates image masking to limit the exploration area of the robotic arm, ensuring that the agent consistently explores regions containing objects. This approach effectively addresses the sparse reward problem and improves the convergence rate of the model. Experimental results from both simulated and real-world environments show that the proposed method accelerates the learning of effective grasping strategies. When image masking is applied, the success rate in the grasping task reaches 80% after 600 iterations. The time required to reach 80% success rate is 25% shorter when image masking is used compared to when it is not used. The main finding of this study is the direct integration of image masking technique with a deep reinforcement learning (DRL) algorithm, which offers significant advancement in robotic arm control. Furthermore, this study shows that image masking technique can substantially reduce training time and improve the object grasping success rate. This innovation enables the robotic arm to better adapt to scenarios that conventional DRL methods cannot handle, thereby improving training efficiency and performance in complex and dynamic industrial applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Reinforcement Learning in Robotics · Soft Robotics and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mechanical Engineering, National Chin-Yi University of Technology, Taichung 41170, Taiwan
Funding
Ministry of Science and Technology in Taiwan
References (48)
Show source-provided references
1. Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J.A., and Goldberg, K. (2017). Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. arXiv.10.15607/RSS.2017.XIII.058 ↗
2. Data-Driven Grasp Synthesis—A Survey10.1109/TRO.2013.2289018 ↗
3. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, Curran Associates Inc.
4. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
5. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015, Proceedings, Part III 18, Springer International Publishing.
6. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
7. Technological development and optimization of pushing and grasping functions in robot arms: A review10.1016/j.measurement.2024.115729 ↗
8. Mastering the game of go without human knowledge10.1038/nature24270 ↗
9. Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2019). Dota 2 with large scale deep reinforcement learning. arXiv.
10. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
11. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv.
12. Berscheid, L., Meißner, P., and Kröger, T. (2019, January 3–8). Robot Learning of Shifting Objects for Grasping in Cluttered Environments. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8968042 ↗
13. Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. (2018, January 1–5). Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593986 ↗
14. Long, J., Shelhamer, E., and Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. arXiv.10.1109/CVPR.2015.7298965 ↗
15. (2022, June 21). Robot Simulator CoppeliaSim: Create, Compose, Simulate, Any Robot—Coppelia Robotics. Available online: https://www.coppeliarobotics.com/.
16. Analysis and Observations From the First Amazon Picking Challenge10.1109/TASE.2016.2600527 ↗
17. Morrison, D., Corke, P., and Leitner, J. (2018). Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach. arXiv.10.15607/RSS.2018.XIV.021 ↗
18. Sundermeyer, M., Mousavian, A., Triebel, R., and Fox, D. (June, January 30). Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xian, China.10.1109/ICRA48506.2021.9561877 ↗
19. Yen-Chen, L., Zeng, A., Song, S., Isola, P., and Lin, T.Y. (August, January 31). Learning to See before Learning to Act: Visual Pretraining for Manipulation. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9197331 ↗
20. Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
21. Efficient Learning of Goal-Oriented Push-Grasping Synergy in Clutter10.1109/LRA.2021.3092640 ↗
22. Learning Target-Oriented Push-Grasping Synergy in Clutter with Action Space Decoupling10.1109/LRA.2022.3204822 ↗
23. Chen, Y., Ju, Z., and Yang, C. (2020, January 19–24). Combining Reinforcement Learning and Rule-based Method to Manipulate Objects in Clutter. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207153 ↗
24. Sarantopoulos, I., Kiatos, M., Doulgeri, Z., and Malassiotis, S. (August, January 31). Split Deep Q-Learning for Robust Object Singulation. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196647 ↗
25. Florence, P., Lynch, C., Zeng, A., Ramirez, O.A., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., and Tompson, J. (2021, January 8–11). Implicit behavioral cloning. Proceedings of the 5th Conference on Robot Learning, London, UK.
26. Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., and Sindhwani, V. (2020, January 16–18). Transporter networks: Rearranging the visual world for robotic manipulation. Proceedings of the Conference on Robot Learning, Virtual.
27. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2018). Densely Connected Convolutional Networks. arXiv.10.1109/CVPR.2017.243 ↗
28. Learning Ambidextrous Robot Grasping Policies10.1126/scirobotics.aau4984 ↗
29. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
30. Rudorfer, M., Suchi, M., Sridharan, M., Vincze, M., and Leonardis, A. (2024, September 05). BURG-Toolkit: Robot Grasping Experiments in Simulation and the Real World. arXiv.org. Available online: https://arxiv.org/abs/2205.14099v1.
31. Han, D., Mulyana, B., Stankovic, V., and Cheng, S. (2023). A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation. Sensors, 23.10.3390/s23073762 ↗
32. Explainability of deep reinforcement learning algorithms in robotic domains by using Layer-wise Relevance Propagation10.1016/j.engappai.2024.109131 ↗
33. Liu, L., Liu, Q., Song, Y., Pang, B., Yuan, X., and Xu, Q. (2021). A Collaborative Control Method of Dual-Arm Robots Based on Deep Reinforcement Learning. Appl. Sci., 11.10.3390/app11041816 ↗
34. A vision-language-guided and deep reinforcement learning-enabled approach for unstructured human-robot collaborative manufacturing task fulfilment10.1016/j.cirp.2024.04.003 ↗
35. Liu, Y., Zhang, H., Wang, X., Li, Q., and Zhao, D. (2023). Reinforcement Learning-Based Robotic Manipulation: A Simulation to Real-World Transfer Approach. Sensors, 23.
36. Chen, L., Wu, M., Liu, Z., and He, J. (2022). Efficient Object Manipulation with Deep Reinforcement Learning: A Simulation Study. Appl. Sci., 12.
37. Zhang, X., Yu, K., Tang, F., and Gao, Y. (2021). Optimizing Robotic Grasping and Pushing Tasks Using Deep Reinforcement Learning in Simulated Environments. Robotics, 10.
38. Influence of fly ash filler on the mechanical properties and water absorption behaviour of epoxy polymer composites reinforced with pineapple leaf fibre for biomedical applications10.1039/D4RA00529E ↗
39. Adopted walking condition for computational simulation approach on bearing of hip joint prosthesis: Review over the past 30 years10.1016/j.heliyon.2022.e12050 ↗
40. Review of machine learning in robotic grasping control in space application10.1016/j.actaastro.2024.04.012 ↗
41. A machine learning-based image processing approach for robotic assembly system10.1016/j.procir.2021.11.152 ↗
42. Deep Reinforcement Learning for Robotic Manipulation with Depth Sensing: A Survey
43. Application of Deep Reinforcement Learning in Robotic Grasping Tasks: Performance Analysis and Improvement Strategies
44. Advanced Hardware Integration for Efficient Learning in Robotic Pushing and Grasping Tasks Using Deep Reinforcement Learning
45. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection10.1177/0278364917710318 ↗
46. Mask R-CNN
47. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
48. Zhu, X., Wang, Y., Dai, J., Yuan, L., and Wei, Y. (2021, January 19–25). Improving Semantic Segmentation via Decoupled Body and Edge Supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.