Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
Source abstract
Abstract
In target-oriented multi-agent tasks, agents collaboratively achieve goals defined by specific objects, or targets, in their environment. The key to success is the effective coordination between agents and these targets, especially in dynamic environments where targets may shift. Agents must adeptly adjust to these changes and re-evaluate their target interactions. Inefficient coordination can lead to resource waste, extended task times, and lower overall performance. Addressing this challenge, we introduce the regulatory hierarchical multi-agent coordination (RHMC), a hierarchical reinforcement learning approach. RHMC divides the coordination task into two levels: a high-level policy, assigning targets based on environmental state, and a low-level policy, executing basic actions guided by individual target assignments and observations. Stabilizing RHMC’s high-level policy is crucial for effective learning. This stability is achieved by reward regularization, reducing reliance on the dynamic low-level policy. Such regularization ensures the high-level policy remains focused on broad coordination, not overly dependent on specific agent actions. By minimizing low-level policy dependence, RHMC adapts more seamlessly to environmental changes, boosting learning efficiency. Testing demonstrates RHMC’s superiority over existing methods in global reward and learning efficiency, highlighting its effectiveness in multi-agent coordination.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Distributed Control Multi-Agent Systems · Adaptive Dynamic Programming Control
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication, Guilin University of Electronic Technology, Guilin 541004, China
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
Funding
Guangxi Natural Science Foundation of China · Guangxi Science and Technology Project · National Natural Science Foundation of China
References (49)
Show source-provided references
1. Yun, W.J., Ha, Y.J., Jung, S., and Kim, J. (2021, January 20–22). Autonomous aerial mobility learning for drone-taxi flight control. Proceedings of the 2021 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Republic of Korea.10.1109/ICTC52510.2021.9620751 ↗
2. Wang, X., Krasowski, H., and Althoff, M. (2021, January 11–14). Commonroad-rl: A configurable reinforcement learning environment for motion planning of autonomous vehicles. Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Nagoya, Japan.10.1109/ITSC48978.2021.9564898 ↗
3. A new sensing direction rotation approach to area coverage optimization in directional sensor network10.20965/jaciii.2020.p0206 ↗
4. Mason, F., Chiariotti, F., Zanella, A., and Popovski, P. (2024). Multi-agent reinforcement learning for coordinating communication and control. IEEE Trans. Cogn. Commun. Netw.10.1109/TCCN.2024.3384492 ↗
5. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
6. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21–26). Deterministic policy gradient algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China.
7. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
8. Ye, D., Liu, Z., Sun, M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., and Guo, Q. (2020, January 7–12). Mastering complex control in MOBA games with deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i04.6144 ↗
9. Chebotar, Y., Hausman, K., Lu, Y., Xiao, T., Kalashnikov, D., Varley, J., Irpan, A., Eysenbach, B., Julian, R., and Finn, C. (2021, January 18–24). Actionable models: Unsupervised offline reinforcement learning of robotic skills. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual Event.
10. Chen, H. (2021, January 2–9). Robotic manipulation with reinforcement learning, state representation learning, and imitation learning (student abstract). Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.10.1609/aaai.v35i18.17881 ↗
11. PPO2: Location privacy-oriented task offloading to edge computing using reinforcement learning for intelligent autonomous transport systems10.1109/TITS.2022.3169421 ↗
12. QoS and privacy-aware routing for 5G enabled industrial internet of things: A federated reinforcement learning approach10.1109/TII.2021.3124848 ↗
13. An efficient parallel reinforcement learning approach to cross-layer defense mechanism in industrial control systems
14. Krnjaic, A., Thomas, J.D., Papoudakis, G., Schäfer, L., Børsting, P., and Albrecht, S.V. (2022). Scalable multi-agent reinforcement learning for warehouse logistics with robotic and human co-workers. arXiv.
15. Game of drones: Multi-UAV pursuit-evasion game with online motion planning by deep reinforcement learning10.1109/TNNLS.2022.3146976 ↗
16. 5G multi-RAT URLLC and eMBB dynamic task offloading with MEC resource allocation using distributed deep reinforcement learning10.1109/JIOT.2022.3177425 ↗
17. Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. (2017). Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE, 12.10.1371/journal.pone.0172395 ↗
18. Lowe, R., Wu, Y.I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. (2017, January 4–9). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
19. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
20. Monotonic value function factorisation for deep multi-agent reinforcement learning
21. Hierarchical coordination multi-agent reinforcement learning with spatio-temporal abstraction10.1109/TETCI.2023.3309738 ↗
22. Collaborative dynamic scheduling in a self-organizing manufacturing system using multi-agent reinforcement learning10.1016/j.aei.2024.102646 ↗
23. Hierarchical relationship modeling in multi-agent reinforcement learning for mixed cooperative–competitive environments10.1016/j.inffus.2024.102318 ↗
24. HiSOMA: A hierarchical multi-agent model integrating self-organizing neural networks with multi-agent deep reinforcement learning10.1016/j.eswa.2024.124117 ↗
25. A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning10.1016/j.cose.2024.103871 ↗
26. Limited-budget output consensus for descriptor multiagent systems with energy constraints10.1109/TCYB.2019.2963172 ↗
27. A knowledge-based approach for multiagent collaboration in smart home: From activity recognition to guidance service10.1109/TIM.2019.2895931 ↗
28. College library personalized recommendation system based on hybrid recommendation algorithm10.1016/j.procir.2019.04.126 ↗
29. Pérolat, J., Leibo, J.Z., Zambaldi, V.F., Beattie, C., Tuyls, K., and Graepel, T. (2017, January 4–9). A multi-agent reinforcement learning model of common-pool resource appropriation. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
30. Feehan, G., and Fatima, S.S. (2022, January 3–5). Augmenting reinforcement learning to enhance cooperation in the iterated prisoner’s dilemma. Proceedings of the 14th International Conference on Agents and Artificial Intelligence (ICAART), Vienna, Austria.10.5220/0010787500003116 ↗
31. Li, S., Wu, Y., Cui, X., Dong, H., Fang, F., and Russell, S.J. (February, January 27). Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
32. Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., and Wu, Y. (2021, January 18–24). The surprising effectiveness of MAPPO in cooperative multi-agent games. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual Event.
33. Neighborhood cooperative multiagent reinforcement learning for adaptive traffic signal control in epidemic regions10.1109/TITS.2022.3173490 ↗
34. Foerster, J.N., Chen, R.Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2017). Learning with opponent-learning awareness. arXiv.
35. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V.F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., and Tuyls, K. (2018, January 10–15). Value-decomposition networks for cooperative multi-agent learning based on team reward. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), Stockholm, Sweden.
36. Pessimism meets invariance: Provably efficient offline mean-field multi-agent RL
37. Stigmergic independent reinforcement learning for multiagent collaboration10.1109/TNNLS.2021.3056418 ↗
38. Kim, Y., Nam, W., Kim, H., Kim, J.-H., and Kim, G. (2019, January 10–15). Curiosity-bottleneck: Exploration by distilling task-specific novelty. Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA.
39. Kulkarni, T.D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. (2016, January 5–10). Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
40. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2018). Diversity is all you need: Learning skills without a reward function. arXiv.
41. Learning multi-agent coordination for enhancing target coverage in directional sensor networks
42. Van Otterlo, M., and Wiering, M. (2012). Reinforcement learning and Markov decision processes. Reinforcement Learning: State-of-the-Art, Springer.10.1007/978-3-642-27645-3_1 ↗
43. A survey of actor-critic reinforcement learning: Standard and natural policy gradients10.1109/TSMCC.2012.2218595 ↗
44. Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. (2017, January 4–9). Bridging the gap between value and policy based reinforcement learning. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
45. A reinforcement learning diffusion decision model for value-based decisions10.3758/s13423-018-1554-2 ↗
46. Hierarchical multi-agent reinforcement learning10.1007/s10458-006-7035-4 ↗
47. Yang, J., Borovikov, I., and Zha, H. (2020, January 9–13). Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Auckland, New Zealand.
48. Loo, Y., Gong, C., and Meghjani, M. (2023, January 19–25). A Hierarchical Approach to Population Training for Human-AI Collaboration. Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI), Macau, China.10.24963/ijcai.2023/336 ↗
49. Ibrahim, M., and Fayad, A. (2022). Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning. arXiv.