Environmental-Impact-Based Multi-Agent Reinforcement Learning
Source abstract
Abstract
To promote cooperation and strengthen the individual impact on the collective outcome in social dilemmas, we propose the Environmental-impact Multi-Agent Reinforcement Learning (EMuReL) method where each agent estimates the “environmental impact” of every other agent, that is, the difference in the current environment state compared to the hypothetical environment in the absence of that other agent. Inspired by the inequity aversion model, the agent then compares its own reward with that of its fellows multiplied by their environmental impacts. If its reward exceeds the scaled reward of one of its fellows, the agent takes “social responsibility” toward that fellow by reducing its own reward. Therefore, the less influential an agent is in reaching the current state, the more social responsibility is taken by other agents. Experiments in the Cleanup (resp. Harvest) test environment demonstrated that agents trained based on EMuReL learned to cooperate more effectively and obtained 54% (39%) and 20% (44%) more total rewards while preserving the same cooperation levels compared to when they were trained based on the two state-of-the-art reward reshaping methods: inequity aversion and social influence.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Traffic control and management · Infrastructure Resilience and Vulnerability Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical & Computer Engineering, Isfahan University of Technology, Isfahan 84156-83111, Iran
Department of Mathematics & Statistics, Brock University, St. Catharines, ON L2S 3A1, Canada
Funding
NSERC Discovery Grant
References (34)
Show source-provided references
1. The nature of common-pool resource problems10.1177/1043463190002003005 ↗
2. Pretorius, A., Cameron, S., van Biljon, E., Makkink, T., Mawjee, S., Plessis, J.d., Shock, J., Laterre, A., and Beguir, K. (2020). A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning. arXiv.
3. Chu, T., Chinchali, S., and Katti, S. (2020, January 26–30). Multi-agent reinforcement learning for networked system control. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
4. Deep reinforcement learning framework for autonomous driving10.2352/ISSN.2470-1173.2017.19.AVM-023 ↗
5. Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaśkowski, W. (2016, January 20–23). Vizdoom: A doom-based ai research platform for visual reinforcement learning. Proceedings of the 2016 IEEE Conference on Computational Intelligence and Games (CIG), Santorini, Greece.10.1109/CIG.2016.7860433 ↗
6. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017, January 8–12). Multi-agent reinforcement learning in sequential social dilemmas. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Sao Paulo, Brazil.
7. Sutton, R.S., and Barto, A.G. (1998). Introduction to Reinforcement Learning, MIT Press.10.1109/TNN.1998.712192 ↗
8. Heemskerk, H. (2020). Social Curiosity in Deep Multi-Agent Reinforcement Learning. [Master’s Thesis, Utrecht University].
9. Is multiagent deep reinforcement learning the answer or the question? A brief survey
10. Barto, A.G., and Simsek, O. (2005). Intrinsic motivation for reinforcement learning systems. Proceedings of the Thirteenth Yale Workshop on Adaptive and Learning Systems, Yale University Press.
11. Klyubin, A.S., Polani, D., and Nehaniv, C.L. (2005, January 2–5). Empowerment: A universal agent-centric measure of control. Proceedings of the 2005 IEEE Congress on Evolutionary Computation, Edinburgh, UK.
12. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J.Z., and De Freitas, N. (2019, January 9–15). Social influence as intrinsic motivation for multi-agent deep reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
13. Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A.A. (May, January 30). Large-Scale Study of Curiosity-Driven Learning. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
14. Pathak, D., Agrawal, P., Efros, A.A., and Darrell, T. (2017, January 6–11). Curiosity-driven exploration by self-supervised prediction. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, NSW, Australia.10.1109/CVPRW.2017.70 ↗
15. Hughes, E., Leibo, J.Z., Phillips, M., Tuyls, K., Dueñez-Guzman, E., Castañeda, A.G., Dunning, I., Zhu, T., McKee, K., and Koster, R. (2018, January 3–8). Inequity aversion improves cooperation in intertemporal social dilemmas. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
16. Empathetic decision making in social networks10.1016/j.artint.2019.05.004 ↗
17. Perolat, J., Leibo, J.Z., Zambaldi, V., Beattie, C., Tuyls, K., and Graepel, T. (2017). A multi-agent reinforcement learning model of common-pool resource appropriation. Adv. Neural Inf. Process. Syst., 30, Available online: https://proceedings.neurips.cc/paper_files/paper/2017/hash/2b0f658cbffd284984fb11d90254081f-Abstract.html.
18. Vinitsky, E., Jaques, N., Leibo, J., Castenada, A., and Hughes, E. (2024, January 01). An Open Source Implementation of Sequential Social Dilemma Games. Available online: https://github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182.
19. Littman, M.L. (1994). Markov games as a framework for multi-agent reinforcement learning. Machine Learning Proceedings 1994, Elsevier.10.1016/B978-1-55860-335-6.50027-1 ↗
20. Van Der Wal, J. (1981). Stochastic Dynamic Programming: Successive Approximations and Nearly Optimal Strategies for Markov Decision Processes and Markov Games, Mathematical Centre Tracts, Mathematisch Centrum.
21. The theory of algorithms
22. Dropout: A simple way to prevent neural networks from overfitting
23. Yu, C., Zhang, M., and Ren, F. (2013, January 1–6). Emotional Multiagent Reinforcement Learning in Social Dilemmas. Proceedings of the Prima, Dunedin, New Zealand.10.1007/978-3-642-44927-7_25 ↗
24. Emotional Multiagent Reinforcement Learning in Spatial Social Dilemmas10.1109/TNNLS.2015.2403394 ↗
25. Badia, A.P., Sprechmann, P., Vitvitskyi, A., Guo, Z.D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., and Bolt, A. (2020, January 26–30). Never Give Up: Learning Directed Exploration Strategies. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
26. Turner, A.M., Hadfield-Menell, D., and Tadepalli, P. (2020, January 7–8). Conservative agency via attainable utility preservation. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3375627.3375851 ↗
27. Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S. (2018). Penalizing side effects using stepwise relative reachability. arXiv.
28. Wang, T., Wang, J., Wu, Y., and Zhang, C. (2019). Influence-based multi-agent exploration. arXiv.
29. Social dilemmas: The anatomy of cooperation10.1146/annurev.soc.24.1.183 ↗
30. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
31. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19–24). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
32. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
33. Canese, L., Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., and Spanò, S. (2021). Multi-Agent Reinforcement Learning: A Review of Challenges and Applications. Appl. Sci., 11.10.1038/s41598-021-94691-7 ↗
34. Singh, S., Barto, A.G., and Chentanez, N. (2004, January 13–18). Intrinsically motivated reinforcement learning. Proceedings of the 17th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.10.21236/ADA440280 ↗