Proximal Policy Optimization Through a Deep Reinforcement Learning Framework for Multiple Autonomous Vehicles at a Non-Signalized Intersection
Source abstract
Abstract
Advanced deep reinforcement learning shows promise as an approach to addressing continuous control tasks, especially in mixed-autonomy traffic. In this study, we present a deep reinforcement-learning-based model that considers the effectiveness of leading autonomous vehicles in mixed-autonomy traffic at a non-signalized intersection. This model integrates the Flow framework, the simulation of urban mobility simulator, and a reinforcement learning library. We also propose a set of proximal policy optimization hyperparameters to obtain reliable simulation performance. First, the leading autonomous vehicles at the non-signalized intersection are considered with varying autonomous vehicle penetration rates that range from 10% to 100% in 10% increments. Second, the proximal policy optimization hyperparameters are input into the multiple perceptron algorithm for the leading autonomous vehicle experiment. Finally, the superiority of the proposed model is evaluated using all human-driven vehicle and leading human-driven vehicle experiments. We demonstrate that full-autonomy traffic can improve the average speed and delay time by 1.38 times and 2.55 times, respectively, compared with all human-driven vehicle experiments. Our proposed method generates more positive effects when the autonomous vehicle penetration rate increases. Additionally, the leading autonomous vehicle experiment can be used to dissipate the stop-and-go waves at a non-signalized intersection.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic control and management · Autonomous Vehicle Technology and Safety · Traffic Prediction and Management Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Smart Transportation Lab, Pukyong National University, Busan 48513, Korea
References (45)
Show source-provided references
1. National Highway Traffic Safety Administration (2017, April 26). Traffic Safety Facts 2015: A Compilation of Motor Vehicle Crash Data from the Fatality Analysis Reporting System and the General Estimates System. The Fact Sheets and Annual Traffic Safety Facts Reports, USA, Available online: https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812384.
2. Help or hindrance? The travel, energy and carbon impacts of highly automated vehicles10.1016/j.tra.2015.12.001 ↗
3. Preparing a nation for automated vehicles: Opportunities, barriers and policy recommendations10.1016/j.tra.2015.04.003 ↗
4. Semi-autonomous adaptive cruise control systems10.1109/TVT.2002.800617 ↗
5. Effect of adaptive cruise control systems on mixed traffic flow near an on-ramp10.1016/j.physa.2006.12.017 ↗
6. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data10.1016/j.trc.2014.09.001 ↗
7. Congested traffic states in empirical observations and microscopic simulations10.1103/PhysRevE.62.1805 ↗
8. The Research of Car-Following Model Based on Real-Time Maximum Deceleration
9. A Markovian Decision Process
10. Howard, R.A. (1960). Dynamic Programming and Markov Processes, The M.I.T. Press.
11. Reinforcement Learning: An Introduction10.1109/TNN.1998.712192 ↗
12. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
13. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
14. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016). Benchmarking deep reinforcement learning for continuous control. arXiv.
15. The Arcade Learning Environment: An Evaluation Platform for General Agents10.1613/jair.3912 ↗
16. Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A Physics Engine for Model-Based Control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Institute of Electrical and Electronics Engineers (IEEE).10.1109/IROS.2012.6386109 ↗
17. End-to-end training of deep visuomotor policies
18. Tan, K.L., Poddar, S., Sarkar, S., and Sharma, A. (2019). Deep Reinforcement Learning for Adaptive Traffic Signal Control. Proceedings of the Volume 3, Rapid Fire Interactive Presentations: Advances in Control Systems; Advances in Robotics and Mechatronics; Automotive and Transportation Systems; Motion Planning and Trajectory Tracking; Soft Mechatronic Actuators and Sensors; Unmanned Ground and Aerial Vehicles, ASME International.10.1115/DSCC2019-9076 ↗
19. Gu, J., Fang, Y., Sheng, Z., and Wen, P. (2020). Double Deep Q-Network with a Dual-Agent for Traffic Signal Control. Appl. Sci., 10.10.3390/app10051622 ↗
20. Gregurić, M., Vujić, M., Alexopoulos, C., and Miletić, M. (2020). Application of Deep Reinforcement Learning in Traffic Signal Control: An Overview and Impact of Open Traffic Data. Appl. Sci., 10.10.3390/app10114011 ↗
21. Cooperative Deep Reinforcement Learning for Large-Scale Traffic Grid Signal Control10.1109/TCYB.2019.2904742 ↗
22. Traffic Light Control by Multiagent Reinforcement Learning Systems
23. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
24. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. arXiv.
25. Schulman, J., Levine, S., Moritz, P., Jordan, M.I., and Abbeel, P. (2015). Trust region policy optimization. arXiv.
26. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
27. Ye, F., Cheng, X., Wang, P., and Chan, C.-Y. (2020). Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning. arXiv.10.1109/IV47402.2020.9304668 ↗
28. Wei, H., Liu, X., Mashayekhy, L., and Decker, K. (2019). Mixed-Autonomy Traffic Control with Proximal Policy Optimization. Proceedings of the 2019 IEEE Vehicular Networking Conference (VNC), Institute of Electrical and Electronics Engineers (IEEE).10.1109/VNC48660.2019.9062809 ↗
29. Pomerleau, D.A. (1988). An autonomous land vehicle in a neural network. Adv. Neural Inf. Process. Syst., 1.
30. Wymann, B., Espi’e, E., Guionneau, C., Dimitrakakis, C., Coulom, R., and Sumner, A. (2013, January 01). TORCS, the Open Racing Car Simulator, v1.3.5. Available online: http://www.torcs.org.
31. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. arXiv.
32. Behrisch, M., Bieker, L., Erdmann, J., and Krajzewicz, D. (2011, January 23–28). SUMO—Simulation of Urban MObility: An Overview. Proceedings of the Third International Conference on Advances in System Simulation, Barcelona, Spain.
33. Krajzewicz, D., Hertkorn, G., Feld, C., and Wagner, P. (2002, January 2–4). SUMO (Simulation of Urban MObility): An open-source traffic simulation. Proceedings of the 4th Middle East Symposium on Simulation and Modelling, Dubai, UAE.
34. Recent development and applications of sumo-simulation of urban mobility
35. Wegener, A., Piórkowski, M., Raya, M., Hellbrück, H., Fischer, S., and Hubaux, J. (2008, January 14–17). TraCI: An Interface for Coupling Road Traffic and Network Simulators. Proceedings of the 11th Communications and Networking Simulation Symposium, New York, NY, USA.10.1145/1400713.1400740 ↗
36. Wu, C., Parvate, K., Kheterpal, N., Dickstein, L., Mehta, A., Vinitsky, E., and Bayen, A.M. Framework for Control and Deep Reinforcement Learning in Traffic. Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).10.1109/ITSC.2017.8317694 ↗
37. Vinitsky, E., Kreidieh, A., Le Flem, L., Kheterpal, N., Jang, K., Wu, F., Liaw, R., Liang, E., and Bayen, A.M. (2018, January 29–31). Benchmarks for Reinforcement Learning in Mixed-Autonomy Traffic. Proceedings of the Conference on Robot Learning, Zürich, Switzerland.
38. Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., and Bayen, A.M. (2017). Flow: Architecture and Benchmarking for Reinforcement Learning in Traffic Control. arXiv.
39. Wu, C., Kreidieh, A., Vinitsky, E., and Bayen, A.M. (2017, January 13–15). Emergent behaviors in mixed-autonomy traffic. Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, CA, USA.
40. Kreidieh, A.R., Wu, C., and Bayen, A.M. (2018). Dissipating Stop-and-Go Waves in Closed and Open Networks Via Deep Reinforcement Learning. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Institute of Electrical and Electronics Engineers (IEEE).10.1109/ITSC.2018.8569485 ↗
41. Treiber, M., and Kesting, A. (2013). Traffic Flow Dynamics. Traffic Flow Dynamics: Data, Models and Simulation, Springer Berlin Heidelberg.10.1007/978-3-642-32460-4 ↗
42. Graesser, L., and Keng, W.L. (2019). Foundations of Deep Reinforcement Learning: Theory and Practice in Python, Addison-Wesley Professional. Chapter 7.
43. Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., and Bayen, A.M. (2017). Flow: A Modular Learning Framework for Autonomy in Traffic. arXiv.
44. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gonzalez, J., Goldberg, K., and Stoica, I. (2017). Ray RLlib: A composable and scalable reinforcement learning library. arXiv.
45. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.