Multi-Dimensional Double Deep Dynamic Q-Network with Aligned Q-Fusion for Dual-Ring Barrier Traffic Signal Control
Source abstract
Abstract
Model-free deep reinforcement learning (MFDRL) is well-suited for real-time traffic signal control (RTSC), as it is a sequential decision problem where the environment is difficult to be a priori modeled, but has performance metrics sufficing as rewards. Previous studies have not ideally employed MFDRL systems at typical intersections with a dual-ring barrier phase structure (DRBPS) and second-by-second signal operation. DRBPS allows phases to time flexibly while satisfying signal timing constraints in engineering, making it complicated yet common in real-world applications. This study proposes an MFDRL method, termed MD4-AQF, to address the RTSC problem under DRBPS. The state can be represented as a 4 × 30 × 4 × 4 array. We define action based on “decision point aligning” to produce a consistent action space that controls dual-ring concurrent phases simultaneously. We developed a training algorithm based on a “multi-dimensional Q-network” that reduces the number of learnable actions from 600+ to 52. We designed action selection based on “aligned Q-fusion” to end two lagging phases simultaneously with a shared compromise sub-action. In simulation experiments, MD4-AQF trains an agent to improve average vehicle delay from 135 s to 48 s. It surpasses another MFDRL ablated method by 14%, and a fully actuated conventional method by 19%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic control and management · Traffic Prediction and Management Techniques · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Transportation and Logistics, Dalian University of Technology, Dalian 116024, China
Funding
National Natural Science Foundation of China · Humanities and Social Science Foundation of the Ministry of Education of China
References (51)
Show source-provided references
1. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
2. Urbanik, T., Tanaka, A., Lozner, B., Lindstrom, E., Lee, K., Quayle, S., Beaird, S., Tsoi, S., Ryus, P., and Gettman, D. (2015). NCHRP Report 812: Signal Timing Manual, Transportation Research Board. [2nd ed.].10.17226/22097 ↗
3. Learning to coordinate traffic signals with adaptive network partition10.1109/TITS.2023.3308594 ↗
4. A meta-reinforcement learning algorithm for traffic signal control to automatically switch different reward functions according to the saturation level of traffic flows10.1111/mice.12924 ↗
5. Area-wide traffic signal control based on a deep graph Q-Network (DGQN) trained in an asynchronous manner10.1016/j.asoc.2022.108497 ↗
6. Large-scale traffic signal control using constrained network partition and adaptive deep reinforcement learning10.1109/TITS.2024.3352446 ↗
7. Traffic signal optimization for partially observable traffic system and low penetration rate of connected vehicles10.1111/mice.12897 ↗
8. Intersec2vec-TSC: Intersection representation learning for large-scale traffic signal control10.1109/TITS.2023.3340153 ↗
9. Enhancing robustness of deep reinforcement learning based adaptive traffic signal controllers in mixed traffic environments through data fusion and multi-discrete actions10.1109/TITS.2024.3399066 ↗
10. Causal inference multi-agent reinforcement learning for traffic signal control10.1016/j.inffus.2023.02.009 ↗
11. Reinforcement learning for traffic signal control in hybrid action space10.1109/TITS.2023.3344585 ↗
12. CCGN: Centralized collaborative graphical transformer multi-agent reinforcement learning for multi-intersection signal free-corridor10.1016/j.neunet.2023.07.027 ↗
13. Traffic signal control using reinforcement learning based on the teacher-student framework10.1016/j.eswa.2023.120458 ↗
14. An inductive heterogeneous graph attention-based multi-agent deep graph infomax algorithm for adaptive traffic signal control10.1016/j.inffus.2022.08.001 ↗
15. A deep reinforcement learning-based cooperative approach for multi-intersection traffic signal control10.1016/j.engappai.2022.105019 ↗
16. Liu, Y., Luo, G., Yuan, Q., Li, J., Jin, L., Chen, B., and Pan, R. (2023, January 19–25). GPLight: Grouped multi-agent reinforcement learning for large-scale traffic signal control. Proceedings of the 32nd International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/23 ↗
17. Mei, H., Li, J., Shi, B., and Wei, H. (2023, January 19–25). Reinforcement learning approaches for traffic signal control under missing data. Proceedings of the 32nd International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/251 ↗
18. Fang, Z., Zhang, F., Wang, T., Lian, X., and Chen, M. (2022, January 17–21). MonitorLight: Reinforcement learning-based traffic signal control using mixed pressure monitoring. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, New York, NY, USA.10.1145/3511808.3557400 ↗
19. Liang, E., Su, Z., Fang, C., and Zhong, R. (March, January 22). OAM: An option-action reinforcement learning framework for universal multi-intersection control. Proceedings of the 36th AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v36i4.20378 ↗
20. CycLight: Learning traffic signal cooperation with a cycle-level strategy10.1016/j.eswa.2024.124543 ↗
21. Dynamic traffic signal control for heterogeneous traffic conditions using max pressure and reinforcement learning10.1016/j.eswa.2024.124416 ↗
22. Auto-learning communication reinforcement learning for multi-intersection traffic light control10.1016/j.knosys.2023.110696 ↗
23. Distributed signal control of arterial corridors using multi-agent deep reinforcement learning10.1109/TITS.2022.3216203 ↗
24. Urban traffic light control via active multi-agent communication and supply-demand modeling10.1109/TKDE.2021.3130258 ↗
25. Zheng, Q., Xu, H., Chen, J., Zhang, D., Zhang, K., and Tang, G. (2022). Double deep Q-network with dynamic bootstrapping for real-time isolated signal control: A traffic engineering perspective. Appl. Sci., 12.10.3390/app12178641 ↗
26. Reinforcement learning for traffic signal control: Incorporating a virtual mesoscopic model for depicting oversaturated traffic conditions10.1016/j.engappai.2023.107005 ↗
27. Extensible prototype learning for real-time traffic signal control10.1111/mice.12955 ↗
28. Multi-agent deep reinforcement learning collaborative traffic signal control method considering intersection heterogeneity10.1016/j.trc.2024.104663 ↗
29. Traffic signal cycle control with centralized critic and decentralized actors under varying intervention frequencies10.1109/TITS.2024.3462153 ↗
30. Intelligent vehicle pedestrian light (IVPL): A deep reinforcement learning approach for traffic signal control10.1016/j.trc.2022.103991 ↗
31. Managing mixed traffic at signalized intersections: An adaptive signal control and CAV coordination system based on deep reinforcement learning10.1016/j.eswa.2023.121959 ↗
32. Two-layer coordinated reinforcement learning for traffic signal control in traffic network10.1016/j.eswa.2023.121111 ↗
33. A deep reinforcement learning approach to traffic signal control with temporal traffic pattern mining
34. (2021). Traffic Controller Assemblies with NTCIP Requirements, 03.08 ed. (Standard No. NEMA TS 2-2021).
35. Optimizing gate control coordination signal for urban traffic network boundaries using multi-agent deep reinforcement learning10.1016/j.eswa.2024.124627 ↗
36. Enhancing traffic signal control with composite deep intelligence10.1016/j.eswa.2023.123020 ↗
37. Multi-objective traffic signal control using network-wide agent coordinated reinforcement learning10.1016/j.eswa.2023.120535 ↗
38. Deep reinforcement learning for traffic signal control with consistent state and reward design approach10.1016/j.knosys.2023.110440 ↗
39. Meta-learning based spatial-temporal graph attention network for traffic signal control10.1016/j.knosys.2022.109166 ↗
40. Ye, Y., Zhou, Y., Ding, J., Wang, T., Chen, M., and Lian, X. (2023, January 19–25). InitLight: Initial model generation for traffic signal control using adversarial inverse reinforcement learning. Proceedings of the 32nd International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/550 ↗
41. Lin, J., Zhu, Y., Liu, L., Liu, Y., Li, G., and Lin, L. (2023, January 19–25). DenseLight: Efficient control for large-scale traffic signals with dense feedback. Proceedings of the 32nd International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/672 ↗
42. Han, X., Zhao, X., Zhang, L., and Wang, W. (2023, January 6–10). Mitigating action hysteresis in traffic signal control with traffic predictive reinforcement learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Long Beach, CA, USA.10.1145/3580305.3599528 ↗
43. Zhang, L., Wu, Q., Shen, J., Lü, L., Du, B., and Wu, J. (2022, January 17–23). Expression might be enough: Representing pressure and demand for reinforcement learning based traffic signal control. Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA.
44. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
45. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018, January 2–7). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the 32th AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11796 ↗
46. Christodoulou, P. (2019). Soft actor-critic for discrete action settings. arXiv.
47. Tavakoli, A., Pardo, F., and Kormushev, P. (2018, January 2–7). Action branching architectures for deep reinforcement learning. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11798 ↗
48. FHWA (2023). Manual on Uniform Traffic Control Devices for Streets and Highways (MUTCD), Federal Highway Administration. [11th ed.].
49. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10–15). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
50. Roderick, M., MacGlashan, J., and Tellex, S. (2017, January 5–10). Implementing the deep Q-network. Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain.
51. (2022). National Academies of Sciences, Engineering, and Medicine. Highway Capacity Manual: A Guide for Multimodal Mobility Analysis, National Academies Press. [7th ed.].