Multi-Level Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
Source abstract
Abstract
Multi-agent reinforcement learning (MARL) has become more and more popular over recent decades, and the need for high-level cooperation is increasing every day because of the complexity of the real-world environment. However, the multi-agent credit assignment problem that serves as the main obstacle to high-level coordination is still not addressed properly. Though lots of methods have been proposed, none of them have thought to perform credit assignments across multi-levels. In this paper, we aim to propose an approach to learning a better credit assignment scheme by credit assignment across multi-levels. First, we propose a hierarchical model that consists of the manager level and the worker level. The manager level incorporates the dilated Gated Recurrent Unit (GRU) to focus on high-level plans and the worker level uses GRU to execute primitive actions conditioned on high-level plans. Then, one centralized critic is designed for each level to learn each level’s credit assignment scheme. To this end, we construct a novel hierarchical MARL algorithm, named MLCA, which can achieve multi-level credit assignment. We also conduct experiments on three classical and challenging tasks to demonstrate the performance of the proposed algorithm against three baseline methods. The results show that our method gains great performance improvement across all maps that require high-level cooperation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin 150001, China
Funding
The National Natural Science Foundation of China
References (26)
Show source-provided references
1. Deep learning10.1038/nature14539 ↗
2. Reinforcement Learning: An Introduction
3. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
4. Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv.
5. Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., and Mordatch, I. (2020, January 26–30). Emergent tool use from multi-agent autocurricula. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
6. A multi-agent reinforcement learning approach to dynamic service composition10.1016/j.ins.2016.05.002 ↗
7. Prabuchandran, K., AN, H.K., and Bhatnagar, S. (2014, January 8–11). Multi-agent reinforcement learning for traffic signal control. Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China.10.1109/ITSC.2014.6958095 ↗
8. Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. [Ph.D. Thesis, King’s College].
9. Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. (2017). Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE, 12.10.1371/journal.pone.0172395 ↗
10. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., and Tuyls, K. (2018, January 10–15). Value-Decomposition Networks For Cooperative Multi-Agent Learning. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2018), Stockholm, Sweden.
11. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual Multi-Agent Policy Gradients. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI 2018), New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
12. Tan, M. (1993, January 27–29). Multi-Agent Reinforcement Learning: Independent versus Cooperative Agents. Proceedings of the Tenth International Conference on Machine Learning (ICML 1993), Amherst, MA, USA.10.1016/B978-1-55860-307-3.50049-6 ↗
13. Optimal Payoff Functions for Members of Collectives10.1142/S0219525901000188 ↗
14. Tumer, K., and Agogino, A. (2007, January 14–18). Distributed agent-based air traffic flow management. Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2007), Honolulu, HI, USA.10.1145/1329125.1329434 ↗
15. Agogino, A.K., and Tumer, K. (2004, January 19–23). Unifying Temporal and Structural Credit Assignment Problems. Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2004), New York, NY, USA.
16. Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. (2018, January 10–15). Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
17. Son, K., Kim, D., Kang, W.J., Hostallero, D.E., and Yi, Y. (2019, January 9–15). QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. Proceedings of the 36th International Conference on Machine Learning (ICML 2019), Long Beach, CA, USA.
18. Wang, J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. (2020, January 26–30). QPLEX: Duplex Dueling Multi-Agent Q-Learning. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
19. Lowe, R., Wu, Y.I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. (2017, January 4–9). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
20. Zhou, M., Liu, Z., Sui, P., Li, Y., and Chung, Y.Y. (2020, January 6–12). Learning Implicit Credit Assignment for Multi-Agent Actor-Critic. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Online.
21. Wang, Y., Han, B., Wang, T., Dong, H., and Zhang, C. (2020, January 26–30). Dop: Off-policy multi-agent decomposed policy gradients. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
22. Wang, T., Dong, H., Lesser, V., and Zhang, C. (2020). ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. arXiv.
23. Wang, T., Gupta, T., Mahajan, A., Peng, B., Whiteson, S., and Zhang, C. (2020, January 26–30). Rode: Learning roles to decompose multi-agent tasks. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
24. Vezhnevets, A.S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. (2017, January 6–11). Feudal networks for hierarchical reinforcement learning. Proceedings of the 34th International Conference on Machine Learning (ICML 2017), Sydney, NSW, Australia.
25. Yu, F., and Koltun, V. (2020, January 2–4). Multi-scale context aggregation by dilated convolutions. Proceedings of the 4th International Conference on Learning Representations (ICLR 2016), San Juan, Puerto Rico.
26. Samvelyan, M., Rashid, T., de Witt, C.S., Farquhar, G., Nardelli, N., Rudner, T.G.J., Hung, C.M., Torr, P.H.S., Foerster, J., and Whiteson, S. (2019). The StarCraft Multi-Agent Challenge. arXiv.