Deep Reinforcement Learning for Sim-to-Real Robot Navigation with a Minimal Sensor Suite for Beach-Cleaning Applications
Source abstract
Abstract
Autonomous beach-cleaning robots require reliable, low-cost navigation on sand. We study Sim-to-Real transfer of deep reinforcement learning (DRL) policies using a minimal sensor suite—wheel-encoder odometry and a single 2-D LiDAR—on a 30 kg differential-drive platform (Raspberry Pi 4). Two policies, Proximal Policy Optimization (PPO) and a masked-action variant (PPO-Mask), were trained in Gazebo + Gymnasium and deployed on the physical robot without hyperparameter retuning. Field trials on firm sand and on a natural loose-sand beach show that PPO-Mask reduces tracking error versus PPO on firm ground (16.6% ISE reduction; 5.2% IAE reduction) and executes multi-waypoint paths faster (square path: 112.48 s vs. 103.46 s). On beach sand, all waypoints were reached within a 1 m tolerance, with mission times of 115.72 s (square) and 81.77 s (triangle). These results indicate that DRL-based navigation with minimal sensing and low-cost compute is feasible in beach settings.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotic Path Planning Algorithms · Modular Robots and Swarm Intelligence · Underwater Vehicles and Communication Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Escuela de Ingeniería Eléctrica, Pontificia Universidad Católica de Valparaíso (PUCV), Valparaíso 2340025, Chile
Instituto de Física, Pontificia Universidad Católica de Valparaíso (PUCV), Valparaíso 2340025, Chile
Funding
FONDEF · FONDECYT
References (45)
Show source-provided references
1. Parker, L. (2025, September 03). Por qué la Contaminación por Plásticos se Convirtió en una Crisis Mundial. National Geographic LA. Available online: https://www.nationalgeographicla.com/medio-ambiente/2024/04/por-que-la-contaminacion-por-plasticos-se-convirtio-en-una-crisis-mundial.
2. H. Barber & Sons, Inc. (2025, September 03). Limpiadoras de Playa. Available online: https://www.hbarber.com/es/surf-rake-limpiador-de-la-playa/los-modelos/.
3. The Searial Cleaners (2025, September 03). BeBot—Beach Cleaning Robot. Searial Cleaners. Available online: https://searial-cleaners.com/our-cleaners/bebot-the-beach-cleaner.
4. Adaptive LiDAR odometry and mapping for autonomous agricultural mobile robots in unmanned farms10.1016/j.compag.2025.110023 ↗
5. Cai, G.S., Lin, H.Y., and Kao, S.F. (2019, January 13–16). Mobile Robot Localization using GPS, IMU and Visual Odometry. Proceedings of the 2019 International Automatic Control Conference (CACS), Keelung, Taiwan.10.1109/CACS47674.2019.9024731 ↗
6. A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving10.1038/s44172-024-00292-3 ↗
7. Autonomous navigation of mobile robots in unknown environments using off-policy reinforcement learning with curriculum learning10.1016/j.eswa.2024.123202 ↗
8. Quiroga, F., Hermosilla, G., Varas, G., Alonso, F., and Schröder, K. (2024). RL-Based Sim2Real Enhancements for Autonomous Beach-Cleaning Agents. Appl. Sci., 14.10.3390/app14114602 ↗
9. Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A.A.A., Yogamani, S., and Pérez, P. (2021). Deep Reinforcement Learning for Autonomous Driving: A Survey. arXiv.10.1109/TITS.2021.3054625 ↗
10. Weerakoon, K., Sathyamoorthy, A.J., Patel, U., and Manocha, D. (2022, January 23–27). TERP: Reliable Planning in Uneven Outdoor Environments using Deep Reinforcement Learning. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA.10.1109/ICRA46639.2022.9812238 ↗
11. ROS 2 Documentation (2025, September 03). Doc ROS. Available online: https://docs.ros.org/en/humble/index.html.
12. Contaminación Marina (2025, September 03). Oceana. Available online: https://chile.oceana.org/campanas/contaminacion-marina/.
13. Plastic contamination of sandy beaches along the southern Baltic—A one season field survey results10.1016/j.oceano.2022.07.004 ↗
14. Microplastics contamination in the fishes of selected sites in Pasig River and Marikina River in the Philippines10.1016/j.marpolbul.2023.114573 ↗
15. Fatma, M.T., Morshedy, A., Khaled, M., and Salem, M. (2025, September 03). EcoBot: An Autonomous Beach-Cleaning Robot for Environmental Sustainability Applications. Available online: https://www.researchgate.net/publication/391451003_EcoBot_An_Autonomous_Beach-Cleaning_Robot_for_Environmental_Sustainability_Applications.
16. Barathraj, M., Yathunanthanasarma, B., Mahiliny, J., and Jayasekara, A.G.B.P. (2024, January 12). Intelligent Beach Cleaning Robot with Dual Modes of Refuse Collection and YOLO-based Detection. Proceedings of the 2024 4th International Conference on Electrical Engineering (EECon), Colombo, Sri Lanka.10.1109/EECon64470.2024.10841881 ↗
17. (2025, September 03). UmiBeach Robótica Autónoma y Eléctrica. Available online: https://umibeach.es/.
18. Varghese, D., and Mohan, A. (2022, January 16–17). Binman: An Autonomous Beach Cleaning Robot. Proceedings of the 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon), Mysuru, India.10.1109/MysuruCon55714.2022.9972499 ↗
19. Ichimura, T., and Nakajima, S.I. (2016, January 7–10). Development of an autonomous beach cleaning robot “Hirottaro”. Proceedings of the 2016 IEEE International Conference on Mechatronics and Automation, Harbin, China.10.1109/ICMA.2016.7558676 ↗
20. Zhu, J., Yang, Y., and Cheng, Y. (2022). SMURF: A Fully Autonomous Water Surface Cleaning Robot with A Novel Coverage Path Planning Method. J. Mar. Sci. Eng., 10.10.3390/jmse10111620 ↗
21. Zhang, Y., Huang, Z., Chen, C., Wu, X., Xie, S., Zhou, H., Gou, Y., Gu, L., and Ma, M. (2023). A Spiral-Propulsion Amphibious Intelligent Robot for Land Garbage Cleaning and Sea Garbage Cleaning. J. Mar. Sci. Eng., 11.10.3390/jmse11081482 ↗
22. VERO: A vacuum-cleaner-equipped quadruped robot for efficient litter removal10.1002/rob.22350 ↗
23. (2025, September 03). ProjectBB. Available online: https://project.bb/.
24. He, N., Yang, Z., Bu, C., Fan, X., Wu, J., Sui, Y., and Que, W. (2024). Learning Autonomous Navigation in Unmapped and Unknown Environments. Sensors, 24.10.3390/s24185925 ↗
25. Salimpour, S., Peña-Queralta, J., Paez-Granados, D., Heikkonen, J., and Westerlund, T. (2025). Sim-to-Real Transfer for Mobile Robots with Reinforcement Learning: From NVIDIA Isaac Sim to Gazebo and Real ROS 2 Robots. arXiv.
26. Kich, V.A., Kolling, A.H., Jesus, J.C.d., Heisler, G.V., Jacobs, H., Bottega, J.A., Kelbouscas, A.L.d.S., Ohya, A., Grando, R.B., and Drews, P.L.J. (2024). Parallel Distributional Deep Reinforcement Learning for Mapless Navigation of Terrestrial Mobile Robots. arXiv.10.2139/ssrn.4585982 ↗
27. Gao, J., Pang, X., Liu, Q., and Li, Y. (2025). Hierarchical Reinforcement Learning for Safe Mapless Navigation with Congestion Estimation. arXiv.10.1109/ICRA55743.2025.11128867 ↗
28. Josifovski, J., Gu, S., Malmir, M., Huang, H., Auddy, S., Navarro-Guerrero, N., Spanos, C., and Knoll, A. (2025). Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics. arXiv.10.1109/IROS58592.2024.10802060 ↗
29. Zhou, Z., Ren, J., Zeng, Z., Xiao, J., Zhang, X., Guo, X., Zhou, Z., and Lu, H. (2023). A safe reinforcement learning approach for autonomous navigation of mobile robots in dynamic environments. CAAI Trans. Intell. Technol., 1–16.10.1049/cit2.12269 ↗
30. Xiao, W., He, T., Dolan, J., and Shi, G. (2024). Safe Deep Policy Adaptation. arXiv.10.1109/ICRA57147.2024.10611340 ↗
31. Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation10.1109/TRO.2019.2920206 ↗
32. Wabersich, K.P., and Zeilinger, M.N. (2021). A predictive safety filter for learning-based control of constrained nonlinear dynamical systems. arXiv.10.1016/j.automatica.2021.109597 ↗
33. Sun, H., Jiang, H., Zhang, L., Wu, C., and Qian, S. (2025). Multi-robot hierarchical safe reinforcement learning autonomous decision-making strategy based on uniformly ultimate boundedness constraints. Sci. Rep., 15.10.1038/s41598-025-89285-6 ↗
34. Adaptive critic design for safety-optimal FTC of unknown nonlinear systems with asymmetric constrained-input10.1016/j.isatra.2024.09.018 ↗
35. Event-triggered H∞ control for unknown constrained nonlinear systems with application to robot arm10.1016/j.apm.2025.116089 ↗
36. Unsupervised image stitching based on Generative Adversarial Networks and feature frequency awareness algorithm10.1016/j.asoc.2025.113466 ↗
37. micro-ROS Developers (2025, September 29). Micro-ROS: ROS 2 on Microcontrollers. Micro-ROS Project., Available online: https://micro.ros.org/.
38. RPLIDAR S2 (2025, September 03). Slamtec. Available online: https://www.slamtec.com/en/s2.
39. Reza Nourbakhsh, I., and Siegwart, R. (2004). Introduction to Autonomous Mobile Robots, MIT Press.
40. Quiroga, F., Hermosilla, G., Farias, G., Fabregas, E., and Montenegro, G. (2022). Position Control of a Mobile Robot through Deep Reinforcement Learning. Appl. Sci., 12.10.3390/app12147194 ↗
41. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
42. A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
43. Raffin, A., and Contributors, S.B. (2025, September 03). Stable-Baselines3 Docs—Reliable Reinforcement Learning Implementations. Stable Baselines. Available online: https://stable-baselines3.readthedocs.io/en/master/.
44. Raffin, A., and Contributors, S.B. (2025, September 03). Stable Baselines3 Contrib Documentation. SB3-Contrib. Available online: https://sb3-contrib.readthedocs.io/en/master/.
45. Shinners, S.M. (1998). Modern Control System Theory and Design, J. Wiley. [2nd ed.].