Pick and Place Operations in Logistics Using a Mobile Manipulator Controlled with Deep Reinforcement Learning
Source abstract
Abstract
Programming robots to perform complex tasks is a very expensive job. Traditional path planning and control are able to generate point to point collision free trajectories, but when the tasks to be performed are complex, traditional planning and control become complex tasks. This study focused on robotic operations in logistics, specifically, on picking objects in unstructured areas using a mobile manipulator configuration. The mobile manipulator has to be able to place its base in a correct place so the arm is able to plan a trajectory up to an object in a table. A deep reinforcement learning (DRL) approach was selected to solve this type of complex control tasks. Using the arm planner’s feedback, a controller for the robot base is learned, which guides the platform to such a place where the arm is able to plan a trajectory up to the object. In addition the performance of two DRL algorithms ((Deep Deterministic Policy Gradient (DDPG)) and (Proximal Policy Optimisation (PPO)) is compared within the context of a concrete robotic task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Reinforcement Learning in Robotics · Robotic Path Planning Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Autonomous and Intelligent Systems, Fundación Tekniker, Iñaki Goenaga, 5-20600 Eibar, Spain
Faculty of Computer Science, P° Manuel Lardizabal, 1-20018 Donostia-San Sebastián, Spain
Funding
Horizon 2020 Framework Programme
References (42)
Show source-provided references
1. Reinforcement learning in robotics: A survey10.1177/0278364913495721 ↗
2. Li, Y. (arXiv, 2017). Deep reinforcement learning: An overview, arXiv.
3. Dömel, A., Kriegel, S., Kaßecker, M., Brucker, M., Bodenmüller, T., and Suppa, M. (2017). Toward fully autonomous mobile manipulation for industrial environments. Int. J. Adv. Robot. Syst., 14.10.1177/1729881417718588 ↗
4. Nassal, U., Damm, M., and Lüth, T. (1994, January 27–29). A mobile platform supporting a manipulator system for an autonomous robot. Proceedings of the 5th World Conference on Robotics Research, Cambridge, MA, USA.
5. Siciliano, B., and Khatib, O. (2016). Springer Handbook of Robotics, Springer.10.1007/978-3-319-32552-1 ↗
6. Padois, V., Fourquet, J.Y., and Chiron, P. (2006). From robotic arms to mobile manipulation: On coordinated motion schemes. Intelligent Production Machines and Systems, Elsevier.10.1016/B978-008045157-2/50100-0 ↗
7. Integrated task planning and control for mobile manipulators10.1177/0278364903022005004 ↗
8. Berntorp, K., Arzén, K.E., and Robertsson, A. (2012, January 3–5). Mobile manipulation with a kinematically redundant manipulator for a pick-and-place scenario. Proceedings of the 2012 IEEE International Conference on Control Applications (CCA), Dubrovnik, Croatia.10.1109/CCA.2012.6402361 ↗
9. Meeussen, W., Wise, M., Glaser, S., Chitta, S., McGann, C., Mihelich, P., Marder-Eppstein, E., Muja, M., Eruhimov, V., and Foote, T. (2010, January 3–8). Autonomous door opening and plugging in with a personal robot. Proceedings of the 2010 IEEE International Conference on Robotics and Automation (ICRA), Anchorage, AK, USA.10.1109/ROBOT.2010.5509556 ↗
10. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., and Graepel, T. (arXiv, 2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm, arXiv.
11. Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., and Jurafsky, D. (arXiv, 2016). Deep reinforcement learning for dialogue generation, arXiv.10.18653/v1/D16-1127 ↗
12. Yoo, S., Yun, K., Choi, J.Y., Yun, K., and Choi, J. (2017, January 21–26). Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.
13. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection10.1177/0278364917710318 ↗
14. Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerik, M., Lampe, T., Tassa, Y., Erez, T., and Riedmiller, M. (arXiv, 2017). Data-efficient deep reinforcement learning for dexterous manipulation, arXiv.
15. Quillen, D., Jang, E., Nachum, O., Finn, C., Ibarz, J., and Levine, S. (2018, January 21–25). Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.10.1109/ICRA.2018.8461039 ↗
16. Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989385 ↗
17. Chen, Y.F., Everett, M., Liu, M., and How, J.P. (2017, January 24–28). Socially aware motion planning with deep reinforcement learning. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202312 ↗
18. Tai, L., Paolo, G., and Liu, M. (2017, January 24–28). Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202134 ↗
19. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning10.1145/3072959.3073602 ↗
20. Heess, N., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, A., and Riedmiller, M. (arXiv, 2017). Emergence of locomotion behaviours in rich environments, arXiv.
21. Breyer, M., Furrer, F., Novkovic, T., Siegwart, R., and Nieto, J. (arXiv, 2018). Flexible Robotic Grasping with Sim-to-Real Transfer Based Reinforcement Learning, arXiv.
22. Stulp, F., Fedrizzi, A., Beetz, M., Autonomous, I., and Group, S. (2009, January 5–7). Learning and Performing Place-Based Mobile Manipulation. Proceedings of the 2009 IEEE 8th International Conference on Development and Learning, Shanghai, China.10.1109/DEVLRN.2009.5175510 ↗
23. Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An introduction, MIT press.10.1109/TNN.1998.712192 ↗
24. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 17). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan.
25. Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., and Wu, Y. (2019, January 18). OpenAI Baselines. Available online: https://github.com/openai/baselines.
26. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Openai, W.Z. (arXiv, 2016). OpenAI Gym, arXiv.
27. Zamora, I., Gonzalez Lopez, N., Vilches, V.M., Hernández Cordero, A., and Robotics, E. (arXiv, 2017). Extending the OpenAI Gym for Robotics: A Toolkit for Reinforcement Learning Using ROS and Gazebo, arXiv.
28. Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (arXiv, 2017). A brief survey of deep reinforcement learning, arXiv.10.1109/MSP.2017.2743240 ↗
29. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (arXiv, 2017). Proximal policy optimization algorithms, arXiv.
30. Schulman, J., Levine, S., Moritz, P., Jordan, M.I., and Abbeel, P. (arXiv, 2015). Trust Region Policy Optimization, arXiv.
31. On information and sufficiency10.1214/aoms/1177729694 ↗
32. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (arXiv, 2015). High-dimensional continuous control using generalized advantage estimation, arXiv.
33. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (arXiv, 2016). Continuous Control With Deep Reinforcement Learning, arXiv.
34. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21–26). Deterministic Policy Gradient Algorithms. Proceedings of the 31st International Conference on Machine Learning (ICML 2014), Beijing, China.
35. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
36. On the theory of the Brownian motion10.1103/PhysRev.36.823 ↗
37. Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R.Y., Chen, X., Asfour, T., Abbeel, P., and Openai, M.A. (arXiv, 2018). Parameter Space Noise for Exploration, arXiv.
38. Todorov, E., Erez, T., and Tassa, Y. (2012, January 7–12). Mujoco: A physics engine for model-based control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal.10.1109/IROS.2012.6386109 ↗
39. Moveit![ROS topics]10.1109/MRA.2011.2181749 ↗
40. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2–4). TensorFlow: A System for Large-Scale Machine Learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Savannah, GA, USA.
41. Ba, J.L., Kiros, J.R., and Hinton, G.E. (arXiv, 2016). Layer Normalization, arXiv.
42. Ng, A.Y. (2004, January 4–8). Feature selection, L 1 vs. L 2 regularization, and rotational invariance. Proceedings of the Twenty-First International Conference on Machine Learning, Banff, AB, Canada.