Path following for Autonomous Ground Vehicle Using DDPG Algorithm: A Reinforcement Learning Approach
Source abstract
Abstract
The potential of autonomous driving technology to revolutionize the transportation industry has attracted significant attention. Path following, a fundamental task in autonomous driving, involves accurately and safely guiding a vehicle along a specified path. Conventional path-following methods often rely on rule-based or parameter-tuning aspects, which may not be adaptable to complex and dynamic scenarios. Reinforcement learning (RL) has emerged as a promising approach that can learn effective control policies from experience without prior knowledge of system dynamics. This paper investigates the effectiveness of the Deep Deterministic Policy Gradient (DDPG) algorithm for steering control in ground vehicle path following. The algorithm quickly converges and the trained agent achieves stable and fast path following, outperforming three baseline methods. Additionally, the agent achieves smooth control without excessive actions. These results validate the proposed approach’s effectiveness, which could contribute to the development of autonomous driving technology.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Traffic control and management · Reinforcement Learning in Robotics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Program of Intelligence and Control, Cluster of Electronics and Mechanical Engineering, School of Science and Technology, Gunma University, 1-5-1 Tenjin-cho, Kiryu 376-8515, Japan
Ryomo Systems Co., Ltd., Ota 373-0853, Japan
Department of Electronic Engineering, Yangzhou University, Yangzhou 225012, China
References (40)
Show source-provided references
1. A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles10.1109/TIV.2016.2578706 ↗
2. Faulwasser, T., Kern, B., and Findeisen, R. (2009, January 15–18). Model predictive path-following for constrained nonlinear systems. Proceedings of the 48h IEEE Conference on Decision and Control (CDC) Held Jointly with 2009 28th Chinese Control Conference, Shanghai, China.10.1109/CDC.2009.5399744 ↗
3. A Survey of Autonomous Driving: Common Practices and Emerging Technologies10.1109/ACCESS.2020.2983149 ↗
4. Performance limitations in reference tracking and path following for nonlinear systems10.1016/j.automatica.2007.06.030 ↗
5. Deep reinforcement learning for quadrotor path following with adaptive velocity10.1007/s10514-020-09951-8 ↗
6. Coulter, R.C. (1992). Implementation of the Pure Pursuit Path Tracking Algorithm, Carnegie-Mellon UNIV Pittsburgh PA Robotics INST. Technical Report.
7. Amidi, O., and Thorpe, C.E. (1991, January 1). Integrated mobile robot control. Proceedings of the Mobile Robots V. SPIE, Boston, MA, USA.
8. Modelling and control strategies in path tracking control for autonomous ground vehicles: A review of state of the art and challenges10.1007/s10846-016-0442-0 ↗
9. Path Following And Time-Varying Feedback Stabilization of a Wheeled Mobile Robot
10. Stanley: The robot that won the DARPA Grand Challenge10.1002/rob.20147 ↗
11. Zhao, W., Queralta, J.P., and Westerlund, T. (2020, January 1–4). Sim-to-real transfer in deep reinforcement learning for robotics: A survey. Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia.10.1109/SSCI47803.2020.9308468 ↗
12. Li, Y. (2017). Deep reinforcement learning: An overview. arXiv.
13. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
14. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21–26). Deterministic Policy Gradient algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China.
15. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
16. Cheng, X., Zhang, S., Cheng, S., Xia, Q., and Zhang, J. (2022). Path-Following and Obstacle Avoidance Control of Nonholonomic Wheeled Mobile Robot Based on Deep Reinforcement Learning. Appl. Sci., 12.10.3390/app12146874 ↗
17. DDPG-based active disturbance rejection 3D path-following control for powered parafoil under wind disturbances10.1007/s11071-023-08444-4 ↗
18. Ma, R., Wang, Y., Wang, S., Cheng, L., Wang, R., and Tan, M. (2023). Sample-Observed Soft Actor–Critic Learning for Path Following of a Biomimetic Underwater Vehicle. IEEE Trans. Autom. Sci. Eng., 1–10.10.1109/TASE.2023.3264237 ↗
19. Martinsen, A.B., and Lekkas, A.M. (2018, January 22–25). Curved Path Following with Deep Reinforcement Learning: Results from Three Vessel Models. Proceedings of the OCEANS 2018 MTS/IEEE Charleston, Charleston, SC, USA.10.1109/OCEANS.2018.8604829 ↗
20. Methods for calculating relative cross-track error for ASABE/ISO Standard 12188-2 from discrete measurements10.13031/trans.59.11902 ↗
21. Martinsen, A.B. (2018). End-to-End Training for Path Following and Control of Marine Vehicles. [Master’s Thesis, Norwegian University of Science and Technology].
22. Control system design of electric power steering for a full vehicle model with active stabilizer
23. Feedback control of a nonholonomic car-like robot
24. Tateyama, Y., Yamada, H., Noyori, J., Mori, Y., Yamamoto, K., Ogi, T., Nishimura, H., Kitamura, N., and Yashiro, H. (2010, January 12–13). Observation of drivers’ behavior at narrow roads using immersive car driving simulator. Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry, Seoul, Republic of Korea.10.1145/1900179.1900263 ↗
25. Fujimura, Y., Hashimoto, S., and Banjerdpongchai, D. (2019, January 7–9). Design of model predictive control with nonlinear disturbance observer for electric power steering system. Proceedings of the 2019 SICE International Symposium on Control Systems (SICE ISCS), Kumamoto, Japan.10.23919/SICEISCS.2019.8758721 ↗
26. Corke, P.I., and Khatib, O. (2011). Robotics, Vision and Control: Fundamental Algorithms in MATLAB, Springer.
27. Deep reinforcement learning-based controller for path following of an unmanned surface vehicle10.1016/j.oceaneng.2019.04.099 ↗
28. Deep reinforcement learning: A brief survey10.1109/MSP.2017.2743240 ↗
29. An introduction to deep reinforcement learning10.1561/2200000071 ↗
30. Han, D., Mulyana, B., Stankovic, V., and Cheng, S. (2023). A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation. Sensors, 23.10.3390/s23073762 ↗
31. Incremental natural actor–critic algorithms
32. Degris, T., White, M., and Sutton, R.S. (2012). Off-policy actor–critic. arXiv.
33. On the theory of the Brownian motion10.1103/PhysRev.36.823 ↗
34. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7–13). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.123 ↗
35. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
36. (2023, April 12). PyTorch. Available online: https://pytorch.org/.
37. Sakhnevych, A., Arricale, V.M., Bruschetta, M., Censi, A., Mion, E., Picotti, E., and Frazzoli, E. (2021). Investigation on the model-based control performance in vehicle safety critical scenarios with varying tyre limits. Sensors, 21.10.3390/s21165372 ↗
38. Santini, S., Albarella, N., Arricale, V.M., Brancati, R., and Sakhnevych, A. (2021). On-board road friction estimation technique for autonomous driving vehicle-following maneuvers. Appl. Sci., 11.10.3390/app11052197 ↗
39. Obereigner, G., Shorten, R., and del Re, L. (2020, January 8–10). Low tyre particle control. Proceedings of the 2020 24th International Conference on System Theory, Control and Computing (ICSTCC), Sinaia, Romania.10.1109/ICSTCC50638.2020.9259771 ↗
40. Development of tire-wear particle emission measurements for passenger vehicles10.1007/s40825-020-00181-z ↗