Model-Based AUV Path Planning Using Curriculum Learning and Deep Reinforcement Learning on a Simplified Electronic Navigation Chart
Source abstract
Abstract
Deep Reinforcement Learning (DRL)-based algorithms have demonstrated substantial effectiveness in tackling complex control problems for autonomous underwater vehicles (AUVs). This paper attempts to evaluate reinforcement learning (RL)-based methods for AUV trajectory planning by incorporating a model of a vehicle’s full motion. In this study, the agent (AUV) is assumed to have no prior knowledge of the environment in which it navigates. Instead, it only receives inputs from navigation sensors and a simulated sonar. Additionally, in the article, a reward function is proposed and described, along with its optimization process, to elicit the desired behaviors in the underwater vehicle. The models are trained and tested on simplified electronic navigation chart (ENC) maps, followed by a comparative analysis against five effective classical methods for trajectory planning. The proposed solution enables efficient, collision-free route planning for the autonomous underwater vehicle, taking its motion dynamics into account to reach the designated target successfully.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Underwater Vehicles and Communication Systems · Robotic Path Planning Algorithms · Maritime Navigation and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Navigation and Naval Weapon, Polish Naval Academy, 81-127 Gdynia, Poland
Faculty of Mechanical and Electrical Engineering, Polish Naval Academy, 81-127 Gdynia, Poland
References (66)
Show source-provided references
1. Chen, J., Sun, C., and Zhang, A. (June, January 30). Autonomous Navigation for Adaptive Unmanned Underwater Vehicles Using Fiducial Markers. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561419 ↗
2. Guo, J., Wang, J., and Bo, Y. (2023). An Observer-Based Adaptive Neural Network Finite-Time Tracking Control for Autonomous Underwater Vehicles via Command Filters. Drones, 7.10.3390/drones7100604 ↗
3. Wang, F., and Zhao, L. (2023). Coordinated Trajectory Planning for Multiple Autonomous Underwater Vehicles: A Parallel Grey Wolf Optimizer. J. Mar. Sci. Eng., 11.10.3390/jmse11091720 ↗
4. Autonomous Data Collection with Timed Communication Constraints for Unmanned Underwater Vehicles10.1109/LRA.2021.3060709 ↗
5. Directions of Development of the Autonomous Unmanned Underwater Vehicles. A Review10.2478/sjpna-2022-0005 ↗
6. Strama, K., Weber, D., and Renkewitz, H. (2021, January 20–23). Evaluation of Wifi Data Transmission Algorithms for Short Distance Underwater Communication. Proceedings of the OCEANS 2021: San Diego–Porto, San Diego, CA, USA.10.23919/OCEANS44145.2021.9705847 ↗
7. Jiang, J., Tian, W., and Han, G. (2023). A Medium Access Control Protocol Based on Interference Cancellation Graph for AUV-Assisted Internet of Underwater Things. Sustainability, 15.10.3390/su15064876 ↗
8. Mulholland, J., and Smolyaninov, I. (September, January 30). Plasmonic-Surface Electromagnetic Wave Communication for Subsea Asset Inspection. Proceedings of the 2022 Sixth Underwater Communications and Networking Conference (UComms), Lerici, Italy.10.1109/UComms56954.2022.9905693 ↗
9. Inzartsev, A.V. (2009). Guidance Laws for Autonomous Underwater Vehicles. Underwater Vehicles, InTech.
10. Fossen, T.I. (2011). Handbook of Marine Craft Hydrodynamics and Motion Control, Wiley. [1st ed.].10.1002/9781119994138 ↗
11. Nonlinear Output Feedback Control of Underwater Vehicle Propellers Using Feedback Form Estimated Axial Flow Velocity10.1109/48.838987 ↗
12. Kot, R., Szymak, P., Piskur, P., and Naus, K. (2024). A-Star (A*) with Map Processing for the Global Path Planning of Autonomous Underwater and Surface Vehicles Operating in Large Areas. Appl. Sci., 14.10.3390/app14178015 ↗
13. Spatial Curvilinear Path Following Control of Underactuated AUV with Multiple Uncertainties10.1016/j.isatra.2016.12.005 ↗
14. Improved Line-of-Sight Trajectory Tracking Control of under-Actuated AUV Subjects to Ocean Currents and Input Saturation10.1016/j.oceaneng.2019.01.025 ↗
15. Three-Dimensional Formation Control Based on Nonlinear Small Gain Method for Multiple Underactuated Underwater Vehicles10.1016/j.oceaneng.2018.01.032 ↗
16. Sliding Mode Based Neural Adaptive Formation Control of Underactuated AUVs with Leader-Follower Strategy10.1016/j.apor.2019.101971 ↗
17. Robust Time-Varying Formation Control for Underactuated Autonomous Underwater Vehicles with Disturbances under Input Saturation10.1016/j.oceaneng.2019.03.017 ↗
18. Three-Dimensional Coordination Control for Multiple Autonomous Underwater Vehicles10.1109/ACCESS.2019.2915933 ↗
19. Galarza, C., Masmitja, I., Prat, J., and Gomaríz, S. (2016, January 21–24). Design of obstacle detection and avoidance system for Guanay II AUV. Proceedings of the 2016 24th Mediterranean Conference on Control and Automation (MED), Athens, Greece.10.1109/MED.2016.7535959 ↗
20. Li, X., Wang, W., Song, J., and Liu, D. (2019, January 13–15). Path planning for autonomous underwater vehicle in presence of moving obstacle based on three inputs fuzzy logic. Proceedings of the 2019 4th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS), Nagoya, Japan.10.1109/ACIRS.2019.8936029 ↗
21. Yan, S., and Pan, F. (2019, January 22–24). Research on route planning of AUV based on genetic algorithms. Proceedings of the 2019 IEEE International Conference on Unmanned Systems and Artificial Intelligence (ICUSAI), Xi’an, China.10.1109/ICUSAI47366.2019.9124785 ↗
22. Improved genetic algorithm for mobile robot path planning in static environments10.1016/j.eswa.2024.123762 ↗
23. Multi-robot path planning using improved particle swarm optimization algorithm through novel evolutionary operators10.1016/j.asoc.2020.106312 ↗
24. ELOS-based Path Following Control for Underactuated Surface Vehicles with Actuator Dynamics10.1016/j.oceaneng.2019.106139 ↗
25. Cui, R., Ge, S.S., Voon Ee How, B., and Choo, Y.S. (2009, January 12–17). Leader-Follower Formation Control of Underactuated AUVs with Leader Position Measurement. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.
26. Nandy, A., and Biswas, M. (2018). Google’s DeepMind and the Future of Reinforcement Learning. Reinforcement Learning, Apress.10.1007/978-1-4842-3285-9 ↗
27. Dong, H., Ding, Z., and Zhang, S. (2020). Deep Q-Networks. Deep Reinforcement Learning, Springer.10.1007/978-981-15-4095-0 ↗
28. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018, January 2–7). Rainbow: Combining Improvements in Deep Reinforcement Learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11796 ↗
29. Busoniu, L., Babuska, R., and De Schutter, B. (2006, January 5–8). Multi-Agent Reinforcement Learning: A Survey. Proceedings of the 2006 9th International Conference on Control, Automation, Robotics and Vision, Singapore.10.1109/ICARCV.2006.345353 ↗
30. Xiao, Z. (2024). Introduction of Reinforcement Learning (RL). Reinforcement Learning, Springer Nature.10.1007/978-981-19-4933-3 ↗
31. Dong, H., Ding, Z., and Zhang, S. (2020). Deep Reinforcement Learning: Fundamentals, Research and Applications, Springer.10.1007/978-981-15-4095-0 ↗
32. Hasselt, H., Guez, A., and Silver, D. (2015). Deep Reinforcement Learning with Double Q-learning. arXiv.
33. Wen, S., Jiang, Y., Cui, B., Gao, K., and Wang, F. (2022). A Hierarchical Path Planning Approach with Multi-SARSA Based on Topological Map. Sensors, 22.10.3390/s22062367 ↗
34. Path Optimization for Marine Vehicles in Ocean Currents Using Reinforcement Learning10.1007/s00773-015-0355-9 ↗
35. Tu, G.T., and Juang, J.G. (2023). UAV Path Planning and Obstacle Avoidance Based on Reinforcement Learning in 3D Environments. Actuators, 12.10.3390/act12020057 ↗
36. Deep-Reinforcement Learning-Based Route Planning with Obstacle Avoidance for Autonomous Vessels10.1007/s10015-023-00909-4 ↗
37. Deep Reinforcement Learning for Adaptive Path Planning and Control of an Autonomous Underwater Vehicle10.1016/j.apor.2022.103326 ↗
38. Risk-Based Implementation of COLREGs for Autonomous Surface Vehicles Using Deep Reinforcement Learning10.1016/j.neunet.2022.04.008 ↗
39. Yu, R., Shi, Z., Huang, C., Li, T., and Ma, Q. (2017, January 26–28). Deep Reinforcement Learning based optimal trajectory tracking control of autonomous underwater vehicle. Proceedings of the 2017 36th Chinese Control Conference (CCC), Dalian, China.10.23919/ChiCC.2017.8028138 ↗
40. AUV position tracking and trajectory control based on fast-deployed Deep Reinforcement Learning method10.1016/j.oceaneng.2021.110452 ↗
41. Reinforcement Learning-based path tracking for underactuated UUV under intermittent communication10.1016/j.oceaneng.2023.116076 ↗
42. Anderlini, E., Parker, G.G., and Thomas, G. (2019). Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning. Appl. Sci., 9.10.3390/app9173456 ↗
43. Palomeras, N., and Ridao, P. (2024). Autonomous Underwater Vehicle Docking Under Realistic Assumptions Using Deep Reinforcement Learning. Drones, 8.10.3390/drones8110673 ↗
44. AUV 3D docking control using Deep Reinforcement Learning10.1016/j.oceaneng.2023.115021 ↗
45. Resource Scheduling Based on Deep Reinforcement Learning in UAV Assisted Emergency Communication Networks10.1109/TCOMM.2022.3170458 ↗
46. 3D UAV Trajectory Design and Frequency Band Allocation for Energy-Efficient and Fair Communication: A Deep Reinforcement Learning Approach10.1109/TWC.2020.3016024 ↗
47. Multi-Agent Cooperation for Computing Power Scheduling in UAVs Empowered Aerial Computing Systems10.1109/JSAC.2024.3459035 ↗
48. Yuan, J., Wang, H., Zhang, H., Lin, C., Yu, D., and Li, C. (2021). AUV Obstacle Avoidance Planning Based on Deep Reinforcement Learning. J. Mar. Sci. Eng., 9.10.3390/jmse9111166 ↗
49. Path Planning Based on Deep Reinforcement Learning for Autonomous Underwater Vehicles Under Ocean Current Disturbance10.1109/TIV.2022.3153352 ↗
50. Wang, Z., Lu, H., Qin, H., and Sui, Y. (2022). Autonomous Underwater Vehicle Path Planning Method of Soft Actor–Critic Based on Game Training. J. Mar. Sci. Eng., 10.10.3390/jmse10122018 ↗
51. Adaptive Deep Reinforcement Learning for Efficient 3D Navigation of Autonomous Underwater Vehicles10.1109/ACCESS.2024.3508031 ↗
52. Liao, X., Li, L., Huang, C., Zhao, X., and Tan, S. (2024). Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning. Front. Neurorobotics, 18.10.3389/fnbot.2024.1466571 ↗
53. Sun, P., Yang, C., Zhou, X., and Wang, W. (2023). Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization. Sensors, 23.10.3390/s23218864 ↗
54. Zhou, Z., Bao, T., Ding, J., Chen, Y., Jiang, Z., and Zhang, B. (2024). An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle. J. Mar. Sci. Eng., 12.10.3390/jmse12122173 ↗
55. Zhao, J., Wang, P., Li, B., and Bai, C. (2023). A DDPG-Based USV Path-Planning Algorithm. Appl. Sci., 13.10.3390/app131910567 ↗
56. Safe Deep Reinforcement Learning-based adaptive control for USV interception mission10.1016/j.oceaneng.2021.110477 ↗
57. Wang, Y., Wang, W., and Chen, D. (2025). Knowledge-Guided Reinforcement Learning with Artificial Potential Field-Based Demonstrations for Multi-Autonomous Underwater Vehicle Cooperative Hunting. J. Mar. Sci. Eng., 13.10.3390/jmse13030423 ↗
58. Szymak, P. (2022, January 15–22). Low-level control of unmanned marine vehicle past. Proceedings of the Polymer Diagnosis Conference, Male, Italy.
59. Sname, T. (1950). Nomenclature for treating the motion of a submerged body through a fluid. The Society of Naval Architects and Marine Engineers, Technical and Research Bulletin, SNAME.
60. Helgason, B., Leifsson, L., Rikhardsson, I., Thorgilsson, H., and Koziel, S. (2012, January 28–31). Low-speed modeling and simulation of torpedo-shaped AUVs. Proceedings of the International Conference on Informatics in Control, Automation and Robotics, Rome, Italy.
61. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv.
62. Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing Function Approximation Error in Actor-Critic Methods. arXiv.
63. Kuznetsov, A., Shvechikov, P., Grishin, A., and Vetrov, D. (2020). Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. arXiv.
64. (2025, January 29). stable-Baselines3-Contrib: TQC, QR-DQN and Other Improvements. Available online: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib.
65. International Hydrographic Organization (IHO) (2018). S-57 Appendix B.1 Annex a: Use of the Object Catalogue for ENC, 4.1.0 ed., International Hydrographic Organization (IHO).
66. (2024, October 10). Stable-Baselines3. Available online: https://github.com/DLR-RM/stable-baselines3.