Application of Deep Reinforcement Learning in Traffic Signal Control: An Overview and Impact of Open Traffic Data
Source abstract
Abstract
Persistent congestions which are varying in strength and duration in the dense traffic networks are the most prominent obstacle towards sustainable mobility. Those types of congestions cannot be adequately resolved by the traditional Adaptive Traffic Signal Control (ATSC). The introduction of Reinforcement Learning (RL) in ATSC as tackled those types of congestions by using on-line learning, which is based on the trial and error approach. Furthermore, RL is prone to the dimensionality curse related to the state–action space size based on which a non-linear quality function is derived. The Deep Reinforcement Learning (DRL) framework uses Deep Neural Networks (DNN) to digest raw traffic data to approximate the quality function of RL. This paper provides a comprehensive analysis of the most recent DRL approaches used for the ATSC algorithm design. Special emphasis is set to overview of the traffic state representation and multi-agent DRL frameworks applied for the large traffic networks. Best practices are provided for choosing the adequate DRL model, hyper-parameters tuning, and model architecture design. Finally, this paper provides a discussion about the importance of the open traffic data concept for the extensive application of DRL in the real world ATSC.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic control and management · Traffic Prediction and Management Techniques · Transportation Planning and Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Transport and Traffic Sciences, Department of Intelligent Transportation Systems, University of Zagreb, 10000 Zagreb, Croatia
Department of Information and Communication Systems Engineering, University of the Aegean, 83200 Karlovasi, Greece
Funding
Horizon 2020 Framework Programme
References (64)
Show source-provided references
1. A Review of Deep Machine Learning10.4028/www.scientific.net/JERA.24.124 ↗
2. Yau, K.L.A., Qadir, J., Khoo, H.L., Ling, M.H., and Komisarczuk, P. (2017). A Survey on Reinforcement Learning Models and Algorithms for Traffic Signal Control. ACM Comput. Surv., 50.10.1145/3068287 ↗
3. Open Traffic Data for Future Service Innovation: Addressing the Privacy Challenges of Driving Data10.4067/S0718-18762014000300007 ↗
4. Traffic light control using deep policy-gradient and value-function-based reinforcement learning10.1049/iet-its.2017.0153 ↗
5. A Deep Reinforcement Learning Network for Traffic Light Cycle Control10.1109/TVT.2018.2890726 ↗
6. Li, Y. (2017). Deep Reinforcement Learning: An Overview. arXiv.
7. Chin, Y.K., Kow, W.Y., Khong, W.L., Tan, M.K., and Teo, K.T.K. (2012, January 14–16). Q-Learning Traffic Signal Optimization within Multiple Intersections Traffic Network. Proceedings of the 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation, Valetta, Malta.10.1109/EMS.2012.75 ↗
8. Q-learning10.1007/BF00992698 ↗
9. Chu, T., Wang, J., Codecà, L., and Li, Z. (2020). Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, IEEE.10.1109/TITS.2019.2901791 ↗
10. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning10.1007/BF00992696 ↗
11. Expert Level Control of Ramp Metering Based on Multi-Task Deep Reinforcement Learning10.1109/TITS.2017.2725912 ↗
12. Kušić, K., Ivanjko, E., and Gregurić, M. (2018, January 19–22). A Comparison of Different State Representations for Reinforcement Learning Based Variable Speed Limit Control. Proceedings of the 2018 26th Mediterranean Conference on Control and Automation (MED), Zadar, Croatia.10.1109/MED.2018.8442986 ↗
13. Shabestary, S.M.A., and Abdulhai, B. (2018, January 4–7). Deep Learning vs. Discrete Reinforcement Learning for Adaptive Traffic Signal Control. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.10.1109/ITSC.2018.8569549 ↗
14. Traffic signal timing via deep reinforcement learning10.1109/JAS.2016.7508798 ↗
15. Kiran, B.R., Thomas, D.M., and Parakkal, R. (2018). An Overview of Deep Learning Based Methods for Unsupervised and Semi-Supervised Anomaly Detection in Videos. J. Imaging, 4.10.3390/jimaging4020036 ↗
16. Genders, W., and Razavi, S.N. (2016). Using a Deep Reinforcement Learning Agent for Traffic Signal Control. arXiv.
17. Cooperative traffic signal control using Multi-step return and Off-policy Asynchronous Advantage Actor–Critic Graph algorithm10.1016/j.knosys.2019.07.026 ↗
18. Lin, Y., Dai, X., Li, L., and Wang, F. (2018). An Efficient Deep Reinforcement Learning Model for Urban Traffic Control. arXiv.
19. Bhagat, S., Banerjee, H., Ho Tse, Z.T., and Ren, H. (2019). Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges. Robotics, 8.10.3390/robotics8010004 ↗
20. Casas, N. (2017). Deep Deterministic Policy Gradient for Urban Traffic Light Control. arXiv.
21. Gao, J., Shen, Y., Liu, J., Ito, M., and Shiratori, N. (2017). Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network. arXiv.
22. Cooperative Deep Q-Learning With Q-Value Transfer for Multi-Intersection Signal Control10.1109/ACCESS.2019.2907618 ↗
23. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
24. Savarese, P. (2019). On the Convergence of AdaBound and its Connection to SGD. arXiv.
25. Calvo, J.A., and Dusparic, I. (2018, January 6–7). Heterogeneous Multi-Agent Deep Reinforcement Learning for Traffic Lights Control. Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science (AICS), Dublin, Ireland.
26. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized Experience Replay. arXiv.
27. Wang, Z., de Freitas, N., and Lanctot, M. (2015). Dueling Network Architectures for Deep Reinforcement Learning. arXiv.
28. Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events10.1016/j.trc.2017.09.020 ↗
29. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
30. Kalweit, G., Huegle, M., and Boedecker, J. (2019). Off-policy Multi-step Q-learning. arXiv.
31. Asis, K.D., Hernandez-Garcia, J.F., Holland, G.Z., and Sutton, R.S. (2017). Multi-step Reinforcement Learning: A Unifying Algorithm. arXiv.
32. Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O., and Clune, J. (2017). Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning. arXiv.
33. Wu, Y., Tan, H., Jiang, Z., and Ran, B. (2019). ES-CTC: A Deep Neuroevolution Model for Cooperative Intelligent Freeway Traffic Control. arXiv.
34. Tan, T., Bao, F., Deng, Y., Jin, A., Dai, Q., and Wang, J. (2019). Cooperative Deep Reinforcement Learning for Large-Scale Traffic Grid Signal Control. IEEE Trans. Cybern., 1–14.
35. A Review of the Self-Adaptive Traffic Signal Control System Based on Future Traffic Environment
36. Review of road traffic control strategies10.1109/JPROC.2003.819610 ↗
37. Zheng, G., Zang, X., Xu, N., Wei, H., Yu, Z., Gayah, V.V., Xu, K., and Li, Z. (2019). Diagnosing Reinforcement Learning for Traffic Signal Control. arXiv.
38. Zhang, R., Ishikawa, A., Wang, W., Striner, B., and Tonguz, O.K. (2020). Using Reinforcement Learning With Partial Vehicle Detection for Intelligent Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, IEEE.10.1109/TITS.2019.2958859 ↗
39. Urban traffic signal control using reinforcement learning agents10.1049/iet-its.2009.0096 ↗
40. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control10.1007/s10458-008-9062-9 ↗
41. Dusparic, I., Monteil, J., and Cahill, V. (2016, January 1–4). Towards autonomic urban traffic control with collaborative multi-policy reinforcement learning. Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.10.1109/ITSC.2016.7795890 ↗
42. Neural Networks for Real-Time Traffic Signal Control10.1109/TITS.2006.874716 ↗
43. Bakker, B., Whiteson, S., Kester, L., and Groen, F.C.A. (2010). Traffic Light Control by Multiagent Reinforcement Learning Systems. Interactive Collaborative Information Systems, Springer.10.1007/978-3-642-11688-9_18 ↗
44. Van der Pol, E., and Oliehoek, F.A. (2016, January 9). Coordinated Deep Reinforcement Learners for Traffic Light Control. Proceedings of the NIPS’16 Workshop on Learning, Inference and Control of Multi-Agent Systems, Barcelona, Spain.
45. Foerster, J., Nardelli, N., Farquhar, G., Afouras, T., Torr, P.H.S., Kohli, P., and Whiteson, S. (2017). Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. arXiv.
46. Wang, S., Xie, X., Huang, K., Zeng, J., and Cai, Z. (2019). Deep Reinforcement Learning-Based Traffic Signal Control Using High-Resolution Event-Based Data. Entropy, 21.10.3390/e21080744 ↗
47. Zambrano-Martinez, J.L., Calafate, C.T., Soler, D., and Cano, J.C. (2017). Towards Realistic Urban Traffic Experiments Using DFROUTER: Heuristic, Validation and Extensions. Sensors, 17.10.3390/s17122921 ↗
48. Kim, T., and Jeong, O. (2019). Cooperative Traffic Signal Control with Traffic Flow Prediction in Multi-Intersection. Sensors, 20.10.3390/s20010137 ↗
49. Zhang, R., Ishikawa, A., Wang, W., Striner, B., and Tonguz, O. (2018). Intelligent traffic signal control: Using reinforcement learning with partial detection. arXiv.
50. Liang, X., Du, X., Wang, G., and Han, Z. (2018). Deep Reinforcement Learning for Traffic Light Control in Vehicular Networks. arXiv.
51. Vidali, A., Crociani, L., Vizzari, G., and Bandini, S. (2019, January 26). A Deep Reinforcement Learning Approach to Adaptive Traffic Lights Management. Proceedings of the WOA, Parma, Italy.
52. Gu, J., Fang, Y., Sheng, Z., and Wen, P. (2020). Double Deep Q-Network with a Dual-Agent for Traffic Signal Control. Appl. Sci., 10.10.3390/app10051622 ↗
53. Pilot Implementation of Public Transport Priority in The City of Zagreb10.7307/ptt.v27i3.1735 ↗
54. Adaptive Traffic Signal Control: Exploring Reward Definition for Reinforcement Learning10.1016/j.procs.2017.05.327 ↗
55. Liu, M., Deng, J., Deng, U., Ming, X., Northeastern, X.Z., and Wang, W. (2017, January 14). Cooperative Deep Reinforcement Learning for Traffic Signal Control. Proceedings of the UrbComp 2017: The 6th International Workshop on Urban Computing, Halifax, NS, Canada.
56. A survey of video processing techniques for traffic applications10.1016/S0262-8856(03)00004-0 ↗
57. Ferdowsi, A., Challita, U., and Saad, W. (2017). Deep Learning for Reliable Mobile Edge Analytics in Intelligent Transportation Systems. arXiv.
58. A Survey of Actor–Critic Reinforcement Learning: Standard and Natural Policy Gradients10.1109/TSMCC.2012.2218595 ↗
59. Khamis, M.A., and Gomaa, W. (2012, January 12). Enhanced multiagent multi-objective reinforcement learning for urban traffic light control. Proceedings of the 11th International Conference on Machine Learning and Applications, Boca Raton, FL, USA.10.1109/ICMLA.2012.108 ↗
60. Chen, Y.-Y., Lv, Y., Li, Z., and Wang, F. (2016, January 4). Long short-term memory model for traffic congestion prediction with online open data. Proceedings of the IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.
61. Traffic Flow Prediction with Big Data: A Deep Learning Approach
62. A tensor-based method for missing traffic data completion10.1016/j.trc.2012.12.007 ↗
63. Nguyen, H., Kieu, M., Wen, T., and Cai, C. (2018). Deep learning methods in transportation domain: A review. IET Intell. Transp. Syst., 12.10.1049/iet-its.2018.0064 ↗
64. Benefits, Adoption Barriers and Myths of Open Data and Open Government10.1080/10580530.2012.716740 ↗