HAPS-PPO: A Multi-Agent Reinforcement Learning Architecture for Coordinated Regional Control of Traffic Signals in Heterogeneous Road Networks
Source abstract
Abstract
The increasing complexity of urban traffic networks has highlighted the potential of Multi-Agent Reinforcement Learning (MARL) for Traffic Signal Control (TSC). However, most existing MARL methods assume homogeneous observation and action spaces among agents, ignoring the inherent heterogeneity of real-world intersections in topology and signal phasing, which limits their practical applicability. To address this gap, we propose HAPS-PPO (Heterogeneity-Aware Policy Sharing Proximal Policy Optimization), a novel MARL framework for coordinated signal control in heterogeneous road networks. HAPS-PPO integrates two key mechanisms: an Observation Padding Wrapper (OPW) that standardizes varying observation dimensions, and a Dynamic Multi-Strategy Grouping Learning (DMSGL) mechanism that trains dedicated policy heads for agent groups with distinct action spaces, enabling adequate knowledge sharing while maintaining structural correctness. Comprehensive experiments in a high-fidelity simulation environment based on a real-world road network demonstrate that HAPS-PPO significantly outperforms Fixed-time control and mainstream MARL baselines (e.g., MADQN, FMA2C), reducing average delay time by up to 44.74% and average waiting time by 59.60%. This work provides a scalable and plug-and-play solution for deploying MARL in realistic, heterogeneous traffic networks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic control and management · Traffic Prediction and Management Techniques · Autonomous Vehicle Technology and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Traffic and Transportation Engineering, Xinjiang University, Urumqi 830017, China
Xinjiang Key Laboratory of Green Construction and Smart Traffic Control of Transportation Infrastructure, Xinjiang University, Urumqi 830017, China
Funding
Xinjiang Uygur Autonomous Region “Tianchi Talent” Introduction Program
References (48)
Show source-provided references
1. Schrank, D., Eisele, B., and Lomax, T. (2021). 2021 Urban Mobility Report, Texas A&M Transportation Institute.
2. Wang, Z., Xu, L., and Ma, J. (2023). Carbon Dioxide Emission Reduction-Oriented Optimal Control of Traffic Signals in Mixed Traffic Flow Based on Deep Reinforcement Learning. Sustainability, 15.10.3390/su152416564 ↗
3. Ouyang, Y., Jain, R., and Varaiya, P. (2016, January 27–30). On the Existence of Near-Optimal Fixed Time Control of Traffic Intersection Signals. Proceedings of the Fifty-Fourth Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.10.1109/ALLERTON.2016.7852221 ↗
4. Lost Time and Cycle Length for Actuated Traffic Signal10.3141/2128-16 ↗
5. Enhancing traffic signal control with composite deep intelligence10.1016/j.eswa.2023.123020 ↗
6. The Sydney coordinated adaptive traffic (SCAT) system philosophy and benefits10.1109/T-VT.1980.23833 ↗
7. Hunt, P.B., Robertson, D.I., Bretherton, R.D., and Winton, R.I. (1981). SCOOT—A Traffic Responsive Method of Coordinating Signals, Urban Networks Division, Traffic Engineering Department, Transport and Road Research Laboratory.
8. Manandhar, B., and Joshi, B. (2018, January 25–27). Adaptive traffic light control with statistical multiplexing technique and particle swarm optimization in smart cities. Proceedings of the 2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS), Kathmandu, Nepal.10.1109/CCCS.2018.8586845 ↗
9. Haddad, T.A., Hedjazi, D., and Aouag, S. (2021). An IoT-based adaptive traffic light control algorithm for isolated intersection. Advances in Computing Systems and Applications, Proceedings of the 4th Conference on Computing Systems and Applications, Algiers, Algeria, 20–21 April 2020, Springer International Publishing.10.1007/978-3-030-69418-0_10 ↗
10. A Survey on Reinforcement Learning Models and Algorithms for Traffic Signal Control10.1145/3068287 ↗
11. Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey10.1109/TITS.2020.3008612 ↗
12. Prabuchandran, K.J., Hemanth Kumar, A.N., and Bhatnagar, S. (2014, January 8–11). Multi-agent reinforcement learning for traffic signal control. Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China.10.1109/ITSC.2014.6958095 ↗
13. Zhang, Z., Yang, J., and Zha, H. (2019). Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv.
14. Ouyang, C., Zhan, Z., and Lv, F. (2024). A Comparative Study of Traffic Signal Control Based on Reinforcement Learning Algorithms. World Electr. Veh. J., 15.10.3390/wevj15060246 ↗
15. Multi-Agent Deep Reinforcement Learning for Decentralized Cooperative Traffic Signal Control
16. Chen, Y., Li, C., Yue, W., Zhang, H., and Mao, G. (2021, January 9–12). Engineering A Large-Scale Traffic Signal Control: A Multi-Agent Reinforcement Learning Approach. Proceedings of the IEEE INFOCOM 2021—IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Virtual.10.1109/INFOCOMWKSHPS51825.2021.9484451 ↗
17. Zhang, Y., Su, R., Zhang, Y., and Sun, C. (2017). Modelling and Traffic Signal Control of Heterogeneous Traffic Systems. arXiv.
18. Chen, B., Tan, K., and Li, J. (2024, January 25–27). HiLight: Heterogeneous Traffic Signal Control for Automatic Drive Guidance Based on Multi-Agent Reinforcement Learning. Proceedings of the 2024 8th CAA International Conference on Vehicular Control and Intelligence (CVCI), Chongqing, China.10.1109/CVCI63518.2024.10830198 ↗
19. A survey on deep reinforcement learning approaches for traffic signal control10.1016/j.engappai.2024.108100 ↗
20. Do, H.K., Quynh Dinh, T., Nguyen, M.D., and Hoa Nguyen, T. (2023, January 2–5). Semantic Communication for Partial Observation Multi-Agent Reinforcement Learning. Proceedings of the 2023 IEEE Statistical Signal Processing Workshop (SSP), Hanoi, Vietnam.10.1109/SSP53291.2023.10207979 ↗
21. Managing mixed traffic at signalized intersections: An adaptive signal control and CAV coordination system based on deep reinforcement learning10.1016/j.eswa.2023.121959 ↗
22. Deep Learning on Traffic Prediction: Methods, Analysis, and Future Directions10.1109/TITS.2021.3054840 ↗
23. Wei, H., Zheng, G., Gayah, V., and Li, Z. (2019). A Survey on Traffic Signal Control Methods. arXiv.
24. Urban traffic signal control optimization through Deep Q Learning and double Deep Q Learning: A novel approach for efficient traffic management10.1007/s11042-024-20060-x ↗
25. Intelligent Traffic Light via Policy-based Deep Reinforcement Learning
26. Amato, C. (2024). An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning. arXiv.
27. Zhou, Y., Liu, S., Qing, Y., Chen, K., Zheng, T., Song, J., and Song, M. (2023). Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?. arXiv.
28. Sunehag, P., Lever, G., Gruslys, A., Marian, W., Vinicius, C., Max, Z., Marc, J., Nicolas, L., Joel, S., and Leibo, Z. (2018, January 10–15). Value-Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward. Proceedings of the 17th International Conference on Autonomous Agents and Multi Agent Systems (AAMAS’18), Stockholm, Sweden.
29. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
30. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. arXiv.
31. Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. (December, January 28). The surprising effectiveness of PPO in cooperative multi-agent games. Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS’22), Orleans, LA, USA.
32. An inductive heterogeneous graph attention-based multi-agent deep graph infomax algorithm for adaptive traffic signal control10.1016/j.inffus.2022.08.001 ↗
33. Wei, H., Zheng, G., Yao, H., and Li, Z. (2018, January 19–23). IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD’18), London, UK.10.1145/3219819.3220096 ↗
34. Multi-agent Deep Reinforcement Learning collaborative Traffic Signal Control method considering intersection heterogeneity10.1016/j.trc.2024.104663 ↗
35. IHG-MA: Inductive heterogeneous graph multi-agent reinforcement learning for multi-intersection traffic signal control10.1016/j.neunet.2021.03.015 ↗
36. Zhang, Y., Li, P., Fan, M., and Sartoretti, G. (2024, January 14–18). HeteroLight: A General and Efficient Learning Approach for Heterogeneous Traffic Signal Control. Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, United Arab Emirates.10.1109/IROS58592.2024.10801524 ↗
37. Large-Scale Traffic Signal Control Using Constrained Network Partition and Adaptive Deep Reinforcement Learning10.1109/TITS.2024.3352446 ↗
38. Pritz, P.J., and Leung, K.K. (2025). Belief States for Cooperative Multi-Agent Reinforcement Learning Under Partial Observability. arXiv.
39. Multi-Agent Transfer Reinforcement Learning with Multi-View Encoder for Adaptive Traffic Signal Control10.1109/TITS.2021.3115240 ↗
40. Baseline and early changes in circulating Serum Amyloid A (SAA) predict survival outcomes in advanced non-small cell lung cancer patients treated with Anti-PD-1/PD-L1 monotherapy10.1016/j.lungcan.2021.05.030 ↗
41. Meta-learning based spatial-temporal graph attention network for traffic signal control10.1016/j.knosys.2022.109166 ↗
42. Kong, A.Y., Lu, B.X., Yang, C.Z., and Zhang, D.M. (2022, January 8–12). A Deep Reinforcement Learning Framework with Memory Network to Coordinate Traffic Signal Control. Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China.10.1109/ITSC55140.2022.9921752 ↗
43. Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control10.1109/ACCESS.2023.3275883 ↗
44. Multiple intersections traffic signal control based on cooperative multi-agent reinforcement learning10.1016/j.ins.2023.119484 ↗
45. Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning10.1016/j.trc.2021.103059 ↗
46. Transferable traffic signal control: Reinforcement learning with graph-centric state representation10.1016/j.trc.2021.103321 ↗
47. Reinforcement Learning for Traffic Signal Control in Hybrid Action Space10.1109/TITS.2023.3344585 ↗
48. Ault, J., and Sharon, G. (2021, January 7–10). Reinforcement Learning Benchmarks for Traffic Signal Control. Proceedings of the Thirty-Fifth Conference on Neural Information Processing Systems (NeurIPS 2021) Datasets and Benchmarks Track, Virtual.