Multi-Agent Reinforcement Learning: A Review of Challenges and Applications
Source abstract
Abstract
In this review, we present an analysis of the most used multi-agent reinforcement learning algorithms. Starting with the single-agent reinforcement learning algorithms, we focus on the most critical issues that must be taken into account in their extension to multi-agent scenarios. The analyzed algorithms were grouped according to their features. We present a detailed taxonomy of the main multi-agent approaches proposed in the literature, focusing on their related mathematical models. For each algorithm, we describe the possible application fields, while pointing out its pros and cons. The described multi-agent algorithms are compared in terms of the most important characteristics for multi-agent reinforcement learning applications—namely, nonstationarity, scalability, and observability. We also describe the most common benchmark environments used to evaluate the performances of the considered methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Evolutionary Algorithms and Applications · Metaheuristic Optimization Algorithms Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic Engineering, University of Rome “Tor Vergata”, Via del Politecnico 1, 00133 Rome, Italy
References (68)
Show source-provided references
1. Yang, H., Liu, X.Y., Zhong, S., and Walid, A. (2020). Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy. SSNR.10.2139/ssrn.3690996 ↗
2. End-to-End Training of Deep Visuomotor Policies
3. Konar, A., Chakraborty, I.G., Singh, S.J., Jain, L.C., and Nagar, A.K. (2013). A deterministic improved q-learning for path planning of a mobile robot. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum., 43.10.1109/TSMCA.2012.2227719 ↗
4. Lin, J.L., Hwang, K.S., Jiang, W.C., and Chen, Y.J. (2016). Gait Balance and Acceleration of a Biped Robot Based on Q-Learning. IEEE Access, 4.10.1109/ACCESS.2016.2570255 ↗
5. Panagiaris, N., Hart, E., and Gkatzia, D. (2021). Generating unambiguous and diverse referring expressions. Comput. Speech Lang., 68.10.1016/j.csl.2020.101184 ↗
6. Matta, M., Cardarilli, G., Di Nunzio, L., Fazzolari, R., Giardino, D., Nannarelli, A., Re, M., and Spanò, S. (2019). A reinforcement learning-based QAM/PSK symbol synchronizer. IEEE Access, 7.10.1109/ACCESS.2019.2938390 ↗
7. Multiagent systems: A survey from a machine learning perspective10.1023/A:1008942012299 ↗
8. Zhuang, Y., Hu, Y., and Wang, H. (2000). Scalability of Multiagent Reinforcement Learning. Interactions in Multiagent Systems, World Scientific. Chapter 1.10.1142/9789813208742_0001 ↗
9. Animal Intelligence: An experimental study of the associate processes in animals10.1037/0003-066X.53.10.1125 ↗
10. A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms10.1287/mnsc.28.1.1 ↗
11. The Complexity of Decentralized Control of Markov Decision Processes10.1287/moor.27.4.819.297 ↗
12. Sutton, R.S. (1990). Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming. Machine Learning Proceedings, Morgan Kaufmann.10.1016/B978-1-55860-141-3.50030-4 ↗
13. Mastering Atari, Go, chess and shogi by planning with a learned model10.1038/s41586-020-03051-4 ↗
14. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play10.1126/science.aar6404 ↗
15. Watkins, C. (1989). Learning from Delayed Rewards. [Ph.D. Thesis, University of Cambridge].
16. Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning I: Introduction, MIT Press.10.1109/TNN.1998.712192 ↗
17. Learning to Predict by the Method of Temporal Differences10.1007/BF00115009 ↗
18. Q-learning10.1007/BF00992698 ↗
19. Simple statistical gradient-following algorithms for connectionist reinforcement learning10.1007/BF00992696 ↗
20. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Harley, T., Lillicrap, T.P., Silver, D., and Kavukcuoglu, K. (2016, January 9–24). Asynchronous Methods for Deep Reinforcement Learning. Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA.
21. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
22. Tan, M. (1993, January 27–29). Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents. Proceedings of the Tenth International Conference on Machine Learning, Amherst, MA, USA.10.1016/B978-1-55860-307-3.50049-6 ↗
23. Claus, C., and Boutilier, C. (1998, January 26–30). The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. Proceedings of the Fifteenth National Conference on Artificial Intelligence, Madison, WI, USA.
24. Q-RTS: A real-time swarm intelligence based on multi-agent Q-learning10.1049/el.2019.0244 ↗
25. Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Giardino, D., Matta, M., Nannarelli, A., Re, M., and Spanò, S. (2020, January 1–4). FPGA Implementation of Q-RTS for Real-Time Swarm Intelligence systems. Proceedings of the 2020 Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.10.1109/IEEECONF51394.2020.9443368 ↗
26. Matignon, L., Laurent, G.J., and Le Fort-Piat, N. (November, January 29). Hysteretic Q-learning: An algorithm for Decentralized Reinforcement Learning in Cooperative Multi-Agent Teams. Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA.10.1109/IROS.2007.4399095 ↗
27. Bloembergen, D., Kaisers, M., and Tuyls, K. (2010). Lenient Frequency Adjusted Q-Learning, University of Luxemburg.
28. QD-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations10.1109/TSP.2013.2241057 ↗
29. Zhang, K., Yang, Z., Liu, H., Zhang, T., and Başar, T. (2018). Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents, ML Research Press.
30. Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016). Continuous control with deep reinforcement learning. arXiv.
31. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on International Conference on Machine Learning, ICML’15, Lille, France.
32. Palmer, G., Tuyls, K., Bloembergen, D., and Savani, R. (2018, January 10–15). Lenient Multi-Agent Deep Reinforcement Learning. Proceedings of the 17th International Conference on Autonomous Agents and Multi Agent Systems, Stockholm, Sweden.
33. Foerster, J., Nardelli, N., Farquhar, G., Torr, P., Kohli, P., and Whiteson, S. (2017, January 6–11). Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
34. Hausknecht, M., and Stone, P. (2015, January 12–14). Deep Recurrent Q-Learning for Partially Observable MDPs. Proceedings of the AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents (AAAI-SDMIA15), Arlington, VA, USA.
35. Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
36. Gupta, J.K., Egorov, M., and Kochenderfer, M. (2017, January 8–12). Cooperative Multi-agent Control Using Deep Reinforcement Learning. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, São Paulo, Brazil.10.1007/978-3-319-71682-4_5 ↗
37. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 6–11). Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
38. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009, January 14–18). Curriculum Learning. Proceedings of the 26th Annual International Conference on Machine Learning, ICML’09, Montreal, QC, Canada.10.1145/1553374.1553380 ↗
39. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI 2018, Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
40. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., and Tuyls, K. (2018, January 10–15). Value-Decomposition Networks For Cooperative Multi-Agent Learning Based on Team Reward. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, International Foundation for Autonomous Agents and Multiagent Systems, AAMAS’18, Stockholm, Sweden.
41. Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. (2018, January 10–15). QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholm, Sweden.
42. Sukhbaatar, S., Szlam, A., and Fergus, R. (2016). Learning multiagent communication with backpropagation. Advances in Neural Information Processing Systems, Proceedings of the 30th Annual Conference on Neural Information Processing Systems, NIPS 2016, Barcelona, Spain, 5–10 December 2016, Curran Associates Inc.
43. Foerster, J.N., Assael, Y.M., de Freitas, N., and Whiteson, S. (2016). Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks. arXiv.
44. Sukhbaatar, S., Szlam, A., Synnaeve, G., Chintala, S., and Fergus, R. (2016). MazeBase: A Sandbox for Learning from Games, Cornell University Library.
45. Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Sasha Vezhnevets, A., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). StarCraft II: A New Challenge for Reinforcement Learning. arXiv.
46. (2021, April 10). MuJoCo: Advanced Physics Simulation. Available online: http://www.mujoco.org.
47. Pham, H.X., La, H.M., Feil-Seifer, D., and Nefian, A. (2018). Cooperative and Distributed Reinforcement Learning of Drones for Field Coverage. arXiv.
48. Joint Optimization of Multi-UAV Target Assignment and Path Planning Based on Multi-Agent Reinforcement Learning10.1109/ACCESS.2019.2943253 ↗
49. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 4–9). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Long Beach, CA, USA.
50. Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks10.1109/TWC.2019.2935201 ↗
51. College Admissions and the Stability of Marriage10.1080/00029890.1962.11989827 ↗
52. Shamsoshoara, A., Khaledi, M., Afghah, F., Razi, A., and Ashdown, J. (2019, January 11–14). Distributed Cooperative Spectrum Sharing in UAV Networks Using Multi-Agent Reinforcement Learning. Proceedings of the 2019 16th IEEE Annual Consumer Communications Networking Conference (CCNC), Las Vegas, NV, USA.10.1109/CCNC.2019.8651796 ↗
53. Jung, S., Yun, W.J., Kim, J., and Kim, J.H. (2021). Coordinated Multi-Agent Deep Reinforcement Learning for Energy-Aware UAV-Based Big-Data Platforms. Electronics, 10.10.3390/electronics10050543 ↗
54. Kong, X., Xin, B., Wang, Y., and Hua, G. (2017, January 21–26). Collaborative Deep Reinforcement Learning for Joint Object Search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.748 ↗
55. Prasad, A., and Dusparic, I. (October, January 29). Multi-agent Deep Reinforcement Learning for Zero Energy Communities. Proceedings of the 2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe), Bucharest, Romania.10.1109/ISGTEurope.2019.8905628 ↗
56. Fang, X., Wang, J., Song, G., Han, Y., Zhao, Q., and Cao, Z. (2020). Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling. Energies, 13.10.3390/en13010123 ↗
57. Nash Q-Learning for General-Sum Stochastic Games
58. Roesch, M., Linder, C., Zimmermann, R., Rudolf, A., Hohmann, A., and Reinhart, G. (2020). Smart Grid for Industry Using Multi-Agent Reinforcement Learning. Appl. Sci., 10.10.3390/app10196900 ↗
59. Shalev-Shwartz, S., Shammah, S., and Shashua, A. (2016). Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving. arXiv.
60. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning10.1016/S0004-3702(99)00052-1 ↗
61. Lin, K., Zhao, R., Xu, Z., and Zhou, J. (2018, January 19–23). Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219993 ↗
62. Calvo, A., and Dusparic, I. (2018, January 6–7). Heterogeneous Multi-Agent Deep Reinforcement Learning for Traffic Lights Control. Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science, Dublin, Ireland.
63. Recent Development and Applications of SUMO –Simulation of Urban Mobility
64. Sui, Y., and Song, S. (2020). A Multi-Agent Reinforcement Learning Framework for Lithium-ion Battery Scheduling Problems. Energies, 13.10.3390/en13081982 ↗
65. Kim, H., and Shin, K.G. (2009, January 1–4). Scheduling of Battery Charge, Discharge, and Rest. Proceedings of the 2009 30th IEEE Real-Time Systems Symposium, Washington, DC, USA.10.1109/RTSS.2009.38 ↗
66. Perolat, J., Leibo, J.Z., Zambaldi, V., Beattie, C., Tuyls, K., and Graepel, T. (2017). A multi-agent reinforcement learning model of common-pool resource appropriation. arXiv.
67. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017, January 8–12). Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS’17, São Paulo, Brazil.
68. Pandey, B. (2016). Adaptive Learning for Mobile Network Management. G2 pro Gradu, Diplomityö. [Master’s Thesis, University of Aalto].