Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
Source abstract
Abstract
This paper investigates the solution to a mobile-robot exploration problem following autonomous driving principles. The exploration task is formulated in this study as a process of building a map while a robot moves in an indoor environment beginning from full uncertainties. The sequence of robot decisions of how to move defines the strategy of the exploration that this paper aims to investigate, applying one of the Deep Reinforcement Learning methods, known as the Deep Deterministic Policy Gradient (DDPG) algorithm. A custom environment is created representing the mapping process with a map visualization, a robot model, and a reward function. The actor-critic network receives and sends input and output data, respectively, to the custom environment. The input is the data from the laser sensor, which is equipped on the robot. The output is the continuous actions of the robot in terms of linear and angular velocities. The training results of this study show the strengths and weaknesses of the DDPG algorithm for the robotic mapping problem. The implementation was developed in MATLAB platform using its corresponding toolboxes. A comparison with another exploration algorithm is also provided.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Robotic Path Planning Algorithms · Optimization and Search Problems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering, Yeungnam University, Gyeongsan 38541, Korea
References (63)
Show source-provided references
1. Lluvia, I., Lazkano, E., and Ansuategi, A. (2021). Active mapping and robot exploration: A survey. Sensors, 21.10.3390/s21072445 ↗
2. Lin, H.Y., and Huang, Y.C. (2021). Collaborative complete coverage and path planning for multi-robot exploration. Sensors, 21.10.3390/s21113709 ↗
3. Shin, F.A.J., and Jang, S.B.H. (2021, January 12–15). Development of Autonomous Navigation Performance Criteria and Related Test Methods for Autonomous Mobile Robot in the Outdoor Environment. Proceedings of the 2021 21st International Conference on Control, Automation and Systems (ICCAS), Jeju, Korea.
4. A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain10.1109/LRA.2021.3093551 ↗
5. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments10.1109/LRA.2019.2891991 ↗
6. The current state and future outlook of rescue robotics10.1002/rob.21887 ↗
7. Towards integrated autonomous underwater operations for ocean mapping and monitoring10.1016/j.arcontrol.2016.09.013 ↗
8. Hong, S., Shyam, P., Bangunharcana, A., and Shin, H. (2022). Robotic Mapping Approach under Illumination-Variant Environments at Planetary Construction Sites. Remote Sens., 14.10.3390/rs14041027 ↗
9. Sun, Z., Wu, B., Xu, C.Z., Sarma, S.E., Yang, J., and Kong, H. (2020–24, January 24). Frontier detection and reachability analysis for efficient 2D graph-slam based active exploration. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341735 ↗
10. Recurrent-octomap: Learning state-based map refinement for long-term semantic mapping with 3-d-lidar data10.1109/LRA.2018.2856268 ↗
11. Lin, J., and Zhang, F. (2019). A fast, complete, point cloud based loop closure for lidar odometry and mapping. arXiv.
12. Object goal navigation using goal-oriented semantic exploration
13. Wurm, K.M., Stachniss, C., and Burgard, W. (2008, January 22–26). Coordinated multi-robot exploration using a segmentation of the environment. Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice, France.10.1109/IROS.2008.4650734 ↗
14. Coordinated Multi-Robot Exploration10.1109/TRO.2004.839232 ↗
15. Hybrid Stochastic Exploration Using Grey Wolf Optimizer and Coordinated Multi-Robot Exploration Algorithms10.1109/ACCESS.2019.2894524 ↗
16. Mobile robots exploration through cnn-based reinforcement learning10.1186/s40638-016-0055-x ↗
17. Tai, L., and Liu, M. (2016). Towards cognitive exploration through deep reinforcement learning for mobile robots. arXiv.10.1186/s40638-016-0055-x ↗
18. Xu, X., Zhang, L., Yang, J., Cao, C., Wang, W., Ran, Y., Tan, Z., and Luo, M. (2022). A Review of Multi-Sensor Fusion SLAM Systems Based on 3D LIDAR. Remote Sens., 14.10.3390/rs14122835 ↗
19. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras10.1109/TRO.2017.2705103 ↗
20. Gautam, A., and Mohan, S. (2012, January 6–9). A review of research in multi-robot systems. Proceedings of the 2012 IEEE 7th international conference on industrial and information systems (ICIIS), Chennai, India.10.1109/ICIInfS.2012.6304778 ↗
21. Waypoint Mobile Robot Exploration Based on Biologically Inspired Algorithms10.1109/ACCESS.2020.3030963 ↗
22. Webster, C., and Ivanov, S. (2020). Robotics, artificial intelligence, and the evolving nature of work. Digital Transformation in Business and Society, Palgrave Macmillan.10.1007/978-3-030-08277-2_8 ↗
23. Machine learning algorithms-a review
24. Deep learning10.1038/nature14539 ↗
25. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
26. Probalistic robotics10.1108/03684920610675292 ↗
27. Alexandre, F., Dominey, P.F., Gaussier, P., Girard, B., Khamassi, M., and Rougier, N.P. (2020). When Artificial Intelligence and Computational Neuroscience meet. A Guided Tour of Artificial Intelligence Research, Springer.10.1007/978-3-030-06170-8_9 ↗
28. A brief history of artificial intelligence: On the past, present, and future of artificial intelligence10.1177/0008125619864925 ↗
29. Tai, L., and Liu, M. (2016). Deep-learning in mobile robotics-from perception to control systems: A survey on why and why not. arXiv.
30. Autonomous exploration of mobile robots through deep neural networks10.1177/1729881417703571 ↗
31. Deep learning of structured environments for robot search10.1007/s10514-018-09821-4 ↗
32. McCormac, J., Handa, A., Davison, A., and Leutenegger, S. (June, January 29). Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and automation (ICRA), Singapore.10.1109/ICRA.2017.7989538 ↗
33. Predicting occupancy distributions of walking humans with convolutional neural networks10.1109/LRA.2018.2800780 ↗
34. The limits and potentials of deep learning for robotics10.1177/0278364918770733 ↗
35. Trajectory optimization using reinforcement learning for map exploration10.1177/0278364907087426 ↗
36. Tai, L., and Liu, M. (2016, January 6–10). A robot exploration strategy based on q-learning network. Proceedings of the 2016 IEEE International Conference on Real-Time Computing and Robotics (RCAR), Angkor Wat, Cambodia.10.1109/RCAR.2016.7784001 ↗
37. Zhelo, O., Zhang, J., Tai, L., Liu, M., and Burgard, W. (2018). Curiosity-driven exploration for mapless navigation with deep reinforcement learning. arXiv.
38. Tai, L., Paolo, G., and Liu, M. (2017, January 24–28). Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202134 ↗
39. Jin, J., Nguyen, N.M., Sakib, N., Graves, D., Yao, H., and Jagersand, M. (August, January 31). Mapless navigation among dynamics with social-safety-awareness: A reinforcement learning approach from 2d laser scans. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9197148 ↗
40. End-to-end navigation strategy with deep reinforcement learning for mobile robots10.1109/TII.2019.2936167 ↗
41. Lu, Z., and Huang, R. (2021, January 15–19). Autonomous mobile robot navigation in uncertain dynamic environments based on deep reinforcement learning. Proceedings of the 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR), Xining, China.10.1109/RCAR52367.2021.9517635 ↗
42. Chen, F., Martin, J.D., Huang, Y., Wang, J., and Englot, B. (2020–24, January 24). Autonomous exploration under uncertainty via deep reinforcement learning on graphs. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341657 ↗
43. Deep reinforcement learning-based automatic exploration for navigation in unknown environment10.1109/TNNLS.2019.2927869 ↗
44. Surmann, H., Jestel, C., Marchel, R., Musberg, F., Elhadj, H., and Ardani, M. (2020). Deep reinforcement learning for real autonomous mobile robot navigation in indoor environments. arXiv.
45. Zhang, J., Tai, L., Liu, M., Boedecker, J., and Burgard, W. (2017). Neural slam: Learning to explore with external memory. arXiv.
46. Xiang, J., Li, Q., Dong, X., and Ren, Z. (2019, January 22–24). Continuous control with deep reinforcement learning for mobile robot navigation. Proceedings of the 2019 Chinese Automation Congress (CAC), Hangzhou, China.10.1109/CAC48633.2019.8996652 ↗
47. Modular deep reinforcement learning from reward and punishment for robot navigation10.1016/j.neunet.2020.12.001 ↗
48. Zhang, J., Springenberg, J.T., Boedecker, J., and Burgard, W. (2017, January 24–28). Deep reinforcement learning with successor features for navigation across similar environments. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8206049 ↗
49. A novel mobile robot navigation method based on deep reinforcement learning10.1177/1729881420921672 ↗
50. Deep reinforcement learning based mobile robot navigation: A review10.26599/TST.2021.9010012 ↗
51. Kollmitz, M., Koller, T., Boedecker, J., and Burgard, W. (2020–24, January 24). Learning human-aware robot navigation from physical interaction via inverse reinforcement learning. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9340865 ↗
52. Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., and Farhadi, A. (June, January 29). Target-driven visual navigation in indoor scenes using deep reinforcement learning. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989381 ↗
53. Xin, J., Zhao, H., Liu, D., and Li, M. (2017, January 20–22). Application of deep reinforcement learning in mobile robot path planning. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.10.1109/CAC.2017.8244061 ↗
54. He, Z., Wang, J., and Song, C. (2021). A review of mobile robot motion planning methods: From classical motion planning workflows to reinforcement learning-based architectures. arXiv.
55. Niu, H., Ji, Z., Arvin, F., Lennox, B., Yin, H., and Carrasco, J. (2021, January 11–14). Accelerated sim-to-real deep reinforcement learning: Learning collision avoidance from human player. Proceedings of the 2021 IEEE/SICE International Symposium on System Integration (SII), Iwaki, Japan.10.1109/IEEECONF49454.2021.9382693 ↗
56. Song, H., Li, A., Wang, T., and Wang, M. (2021). Multimodal Deep Reinforcement Learning with Auxiliary Task for Obstacle Avoidance of Indoor Mobile Robot. Sensors, 21.10.3390/s21041363 ↗
57. Feng, S., Sebastian, B., and Ben-Tzvi, P. (2021). A collision avoidance method based on deep reinforcement learning. Robotics, 10.10.3390/robotics10020073 ↗
58. Multi-goal Visual Navigation with Collision Avoidance via Deep Reinforcement Learning10.1109/TIM.2022.3158384 ↗
59. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
60. (2022, May 18). Available online: https://www.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html.
61. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
62. (2022, May 18). Available online: https://www.mathworks.com/help/reinforcement-learning/ug/create-custom-matlab-environment-from-template.html.
63. (2022, July 23). YouTube Video. Available online: https://youtu.be/SS1h7hn9ZBE.