A Comparative Study of Deep Reinforcement Learning Algorithms for Urban Autonomous Driving: Addressing the Geographic and Regulatory Challenges in CARLA
Source abstract
Abstract
To enable autonomous driving in real-world environments that involve a diverse range of geographic variations and complex traffic regulations, it is essential to investigate Deep Reinforcement Learning (DRL) algorithms capable of policy learning in high-dimensional environments characterized by intricate state–action interactions. In particular, closed-loop experiments, which involve continuous interaction between an agent and their driving environment, serve as a critical framework for improving the practical applicability of DRL algorithms in autonomous driving systems. This study empirically analyzes the capabilities of several representative DRL algorithms—namely DDPG, SAC, TD3, PPO, TQC, and CrossQ—in handling various urban driving scenarios using the CARLA simulator within a closed-loop framework. To evaluate the adaptability of each algorithm to geographical variability and complex traffic laws, scenario-specific reward and penalty functions were carefully designed and incorporated. For a comprehensive performance assessment of the DRL algorithms, we defined several driving performance metrics, including Route Completion, Centerline Deviation Mean, Episode Reward Mean, and Success Rate, which collectively reflect the quality of the driving in terms of its completeness, stability, efficiency, and comfort. Experimental results demonstrate that TQC and SAC, both of which adopt off-policy learning and stochastic policies, achieve superior sample efficiency and learning performances. Notably, the presence of geographically variant features—such as traffic lights, intersections, and roundabouts—and their associated traffic rules within a given town pose significant challenges to driving performance, particularly in terms of Route Completion, Success Rate, and lane-keeping stability. In these challenging scenarios, the TQC algorithm achieved a Route Completion rate of 0.91, substantially outperforming the 0.23 rate observed with DDPG. This performance gap highlights the advantage of approaches like TQC and SAC, which address Q-value overestimation through statistical methods, in improving the robustness and effectiveness of autonomous driving in diverse urban environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic control and management · Autonomous Vehicle Technology and Safety · Traffic and Road Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Konkuk University, Seoul 05029, Republic of Korea
Korea Electronics Technology Institute, Seongnam 13488, Republic of Korea
Department of Smart Automotive, Soonchunhyang University, Asan 31538, Republic of Korea
Funding
Soonchunhyang University Research Fund
References (82)
Show source-provided references
1. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–8). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS), Lake Tahoe, NV, USA.
2. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
4. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Li, F.-F. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
5. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
6. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 14–19). nuScenes: A multimodal dataset for autonomous driving. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01164 ↗
7. Jun, W., and Lee, S. (2025). A Comparative Study and Optimization of Camera-Based BEV Segmentation for Real-Time Autonomous Driving. Sensors, 25.10.3390/s25072300 ↗
8. Rethinking Real-Time Lane Detection Technology for Autonomous Driving
9. Study on Point Cloud Based 3D Object Detection for Autonomous Driving
10. Jun, W., Yoo, J., and Lee, S. (2024). Synthetic Data Enhancement and Network Compression Technology of Monocular Depth Estimation for Real-Time Autonomous Driving System. Sensors, 24.10.3390/s24134205 ↗
11. Jun, W., and Lee, S. (2023). Optimal Configuration of Multi-Task Learning for Autonomous Driving. Sensors, 23.10.3390/s23249729 ↗
12. Son, M., Won, Y., and Lee, S. (2025). Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques. Appl. Sci., 15.10.3390/app15031430 ↗
13. Son, M., and Lee, S. (2025). Advancing Multimodal Large Language Models: Optimizing Prompt Engineering Strategies for Enhanced Performance. Appl. Sci., 15.10.3390/app15073992 ↗
14. Yang, Z., Jia, X., Li, H., and Yan, J. (2023). LLM4Drive: A Survey of Large Language Models for Autonomous Driving. arXiv.
15. Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., and Liao, K. (2024, January 2–6). A Survey on Multimodal Large Language Models for Autonomous Driving. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACVW60836.2024.00106 ↗
16. Research Trends Focused on End-to-End Learning Technologies for Autonomous Vehicles
17. Ye, F., Zhang, S., Wang, P., and Chan, C. (2021, January 11–13). A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles. Proceedings of the 2021 IEEE Intelligent Vehicles Symposium (IV), Nagoya, Japan.10.1109/IV48863.2021.9575880 ↗
18. A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions10.1109/TITS.2024.3452480 ↗
19. A Survey of Deep RL and IL for Autonomous Driving Policy Learning10.1109/TITS.2021.3134702 ↗
20. Multi-agent reinforcement learning for autonomous vehicles: A survey10.1007/s43684-022-00045-z ↗
21. Recent advances in reinforcement learning-based autonomous driving behavior planning: A survey10.1016/j.trc.2024.104654 ↗
22. Ross, S., Gordon, G.J., and Bagnell, J.A. (2011, January 11–13). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS 2011), Fort Lauderdale, FL, USA.
23. Ross, S., and Bagnell, D. (2010, January 13–15). Efficient Reductions for Imitation Learning. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010), Chia Laguna Resort, Italy.
24. Imitation Learning from Imperfect Demonstration
25. Codevilla, F., Muller, M., Lopez, A., Koltun, V., and Dosovitskiy, A. (2018, January 21–25). End-to-End Driving via Conditional Imitation Learning. Proceedings of the International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.10.1109/ICRA.2018.8460487 ↗
26. Ho, J., and Ermon, S. (2016, January 5–10). Generative Adversarial Imitation Learning (GAIL). Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS), Barcelona, Spain.
27. End-to-end Autonomous Driving: Challenges and Frontiers10.1109/TPAMI.2024.3435937 ↗
28. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013, January 6–7). Playing Atari with Deep Reinforcement Learning. Proceedings of the NIPS Deep Learning Workshop 2013, Lake Tahoe, NV, USA.
29. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv.
30. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
31. Schulman, J., Levine, S., Moritz, P., Jordan, M.I., and Abbeel, P. (2015, January 6–11). Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France.
32. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous Control with Deep Reinforcement Learning. arXiv.
33. Fujimoto, S., Hoof, H., and Meger, D. (2018, January 10–15). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden.
34. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10–15). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden.
35. Bhatt, A., Rhinehart, N., and Levine, S. (2019). CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. arXiv.
36. Kuznetsov, A., Laskin, M., Kostrikov, I., and Abbeel, P. (2020, January 12–18). Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. Proceedings of the 37th International Conference on Machine Learning (ICML), Vienna, Austria.
37. Modified DDPG car-following model with a real-world human driving experience with CARLA simulator10.1016/j.trc.2022.103987 ↗
38. (2023). Jumman Hossain, Autonomous Driving with Deep Reinforcement Learning in CARLA Simulation. arXiv.
39. Sharma, R., and Garg, P. (2024, January 18–20). Optimizing Autonomous Driving with Advanced Reinforcement Learning: Evaluating DQN and PPO. Proceedings of the 5th International Conference on Smart Electronics and Communication (ICOSEC), Tiruchirappalli, India.10.1109/ICOSEC61587.2024.10722344 ↗
40. Deep reinforcement learning based control for Autonomous Vehicles in CARLA10.1007/s11042-021-11437-3 ↗
41. CARLA: An Open Urban Driving Simulator
42. Pomerleau, D.A. (1988, January 27–30). ALVINN: An Autonomous Land Vehicle in a Neural Network. Proceedings of the 1st International Conference on Neural Information Processing Systems (NIPS), Denver, CO, USA.
43. Bojarski, M., Testa, D.D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., and Zhang, J. (2016). End to End Learning for Self-Driving Cars. arXiv.
44. Rhinehart, N., McAllister, R., and Levine, S. (2018, January 18–22). Deep Imitative Models for Flexible Inference, Planning, and Control. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
45. Kim, S., and Shin, J. (2024, January 18–20). Deep Imitation Learning for End-to-End Autonomous Driving: Integration of Carla and OpenAI Gym. Proceedings of the 5th International Conference on Smart Electronics and Communication (ICOSEC), Tiruchirappalli, India.
46. Shi, J., Zhang, T., Zhan, J., Chen, S., Xin, J., and Zheng, N. (2023, January 4–7). Efficient Lane-changing Behavior Planning via Reinforcement Learning with Imitation Learning Initialization. Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA.10.1109/IV55152.2023.10186577 ↗
47. Lu, Y., Fu, J., Tucker, G., Pan, X., Bronstein, E., Roelofs, R., Sapp, B., White, B., Faust, A., and Whiteson, S. (2022). Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios. arXiv.10.1109/IROS55552.2023.10342038 ↗
48. Nehme, G., and Deo, T.Y. (2023). Safe Navigation: Training Autonomous Vehicles using Deep Reinforcement Learning in CARLA. arXiv.
49. Khlifi, A., Othmani, M., and Kherallah, M. (2025). A Novel Approach to Autonomous Driving Using Double Deep Q-Network-Based Deep Reinforcement Learning. World Electr. Veh. J., 16.10.20944/preprints202501.0617.v1 ↗
50. Terapaptommakol, W., Phaoharuhansa, D., Koowattanasuchat, P., and Rajruangrabin, J. (2022). Design of Obstacle Avoidance for Autonomous Vehicle Using Deep Q-Network and CARLA Simulator. World Electr. Veh. J., 13.10.3390/wevj13120239 ↗
51. Cai, P., Wang, S., Wang, H., and Liu, M. (2021). Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning. arXiv.
52. Elallid, B.B., Benamar, N., Mrani, N., and Rachidi, T. (2022, January 22–23). DQN-based Reinforcement Learning for Vehicle Control of Autonomous Vehicles Interacting With Pedestrians. Proceedings of the 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT), Sakhir, Bahrain.10.1109/3ICT56508.2022.9990801 ↗
53. Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness10.1016/j.trc.2021.103452 ↗
54. CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
55. Činčurak, D., Grbić, R., Vranješ, M., and Vranješ, D. (2024, January 16–18). Autonomous Vehicle Control in CARLA Simulator using Reinforcement Learning. Proceedings of the 66th International Symposium ELMAR-2024, Zadar, Croatia.10.1109/ELMAR62909.2024.10694632 ↗
56. Perez-Gil, O., Barea, R., Lopez-Guillen, E., Bergasa, L.M., Gomez-Huelamo, C., Gutierrez, R., and Diaz, A. (2021, January 11–17). Deep Reinforcement Learning Based Control Algorithms: Training and Validation Using the ROS Framework in CARLA Simulator for Self-Driving Applications. Proceedings of the 2021 IEEE Intelligent Vehicles Symposium (IV), Nagoya, Japan.10.1109/IV48863.2021.9575616 ↗
57. Policy-Based Reinforcement Learning for Training Autonomous Driving Agents in Urban Areas With Affordance Learning10.1109/TITS.2021.3115235 ↗
58. Matsioris, G., Theocharous, A., Tsourveloudis, N., and Doitsidis, L. (2024, January 11–14). Towards developing a framework for autonomous electric vehicles using CARLA: A Validation using the Deep Deterministic Policy Gradient algorithm. Proceedings of the 2024 32nd Mediterranean Conference on Control and Automation (MED), Chania, Greece.10.1109/MED61351.2024.10566221 ↗
59. Deductive Reinforcement Learning for Visual Autonomous Urban Driving Navigation10.1109/TNNLS.2021.3109284 ↗
60. Anzalone, L., Barra, S., and Nappi, M. (2021, January 19–22). Reinforced Curriculum Learning for Autonomous Driving in CARLA. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.10.1109/ICIP42928.2021.9506673 ↗
61. Gutiérrez-Moreno, R., Barea, R., López-Guillén, E., Araluce, J., and Bergasa, L.M. (2022). Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator. Sensors, 22.10.3390/s22218373 ↗
62. Wang, L., Liu, J., Shao, H., Wang, W., Chen, R., Liu, Y., and Waslander, S.L. (2023, January 10–14). Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors. Proceedings of the Robotics: Science and Systems (RSS), Daegu, Republic of Korea.10.15607/RSS.2023.XIX.102 ↗
63. An Empirical Study of DDPG and PPO-Based Reinforcement Learning Algorithms for Autonomous Driving10.1109/ACCESS.2023.3330665 ↗
64. Lane Changing of Autonomous Vehicle Based on TD3 algorithm in Human-machine Hybrid Driving Environment
65. Elallid, B.B., Alaoui, H.E., and Benamar, N. (2023, January 20–23). Deep Reinforcement Learning for Autonomous Vehicle Intersection Navigation. Proceedings of the 2023 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT), Sakhir, Bahrain.
66. Multi-task Safe Reinforcement Learning for Navigating Intersections in Dense Traffic10.1016/j.jfranklin.2022.06.052 ↗
67. Wang, K., She, C., Li, Z., Yu, T., Li, Y., and Sakaguchi, K. (2024). Roadside Units Assisted Localized Automated Vehicle Maneuvering: An Offline Reinforcement Learning Approach. arXiv.10.1109/ITSC58415.2024.10920203 ↗
68. Liu, Q., Dang, F., Wang, X., and Ren, X. (2022, January 8–12). Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller. Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China.10.1109/ITSC55140.2022.9921741 ↗
69. Elallid, B.B., Benamar, N., Bagaa, M., and Hadjadj-Aoul, Y. (2024). Enhancing Autonomous Driving Navigation Using Soft Actor-Critic. Future Internet, 16.10.3390/fi16070238 ↗
70. Aghdasian, A.J., Ardakani, A.H., Aqabakee, K., and Abdollahi, F. (2023, January 13–15). Autonomous Driving using Residual Sensor Fusion and Deep Reinforcement Learning. Proceedings of the 2023 11th RSI International Conference on Robotics and Mechatronics (ICRoM), Tehran, Iran.10.1109/ICRoM60803.2023.10412516 ↗
71. Abdollahian, S.A., Fazel, S., Hadadi, M.R., and Aghdasian, A.J. (2024, January 15–22). Enhancing Autonomous Vehicle Control through Sensor Fusion, NARX-based Reinforcement Learning with Soft Actor-Critic (SAC) in CARLA Simulator. Proceedings of the RoboCup 2024: Robot World Cup XXVII, Cham, Switzerland.10.1007/978-3-031-85859-8_19 ↗
72. Mixture of Experts Framework Based on Soft Actor-Critic algorithm for Highway Decision-Making of Connected and Automated Vehicles10.1186/s10033-024-01158-7 ↗
73. A formal basis for the heuristic determination of minimum cost paths10.1109/TSSC.1968.300136 ↗
74. Feng, Y., Zhang, W., and Zhu, J. (2023). Application of an improved A* algorithm for the path analysis of urban multi-type transportation systems. Appl. Sci., 13.10.3390/app132413090 ↗
75. Intelligent vehicle driving decision-making model based on variational AutoEncoder network and deep reinforcement learning10.1016/j.eswa.2024.126319 ↗
76. End-to-end autonomous driving controller using semantic segmentation and variational autoencoder
77. Diaz-Diaz, A., Aranda, M., Barea, R., Bergasa, L.M., and Arroyo, R. (2022, January 5–9). HD Maps: Exploiting OpenDRIVE Potential for Path Planning and Map Monitoring. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.10.1109/IV51971.2022.9827297 ↗
78. Jung, S., Jun, W., Oh, T., and Lee, S. (2024, January 6–8). Performance Analysis of Motion Planning for Outdoor Autonomous Delivery Robot. Proceedings of the 2024 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.10.1109/ICCE59016.2024.10444300 ↗
79. Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2023). Mastering Diverse Domains through World Models. arXiv.
80. Learning by Playing: Solving Sparse Reward Tasks from Scratch
81. Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. (Int. J. Robot. Res., 2024). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, Int. J. Robot. Res., Online First.10.1177/02783649241273668 ↗
82. Reed, S., Żołna, K., Parisotto, E., Colmenarejo, S.G., Novikov, A., Barth-Maron, G., Giménez, M., Sulsky, Y., Kay, J., and Springenberg, J.T. (2022). A Generalist Agent. arXiv.