An Effective Training Method for Counterfactual Multi-Agent Policy Network Based on Differential Evolution Algorithm
Source abstract
Abstract
Due to the advantages of a centralized critic to estimate the Q-function value and decentralized actors to optimize the agents’ policies, counterfactual multi-agent (COMA) stands out in most multi-agent reinforcement learning (MARL) algorithms. The sharing of policy parameters can improve sampling efficiency and learning effectiveness, but it may lead to a lack of policy diversity. Hence, to balance parameter sharing and diversity among agents in COMA has been a persistent research topic. In this paper, an effective training method for a COMA policy network based on a differential evolution (DE) algorithm is proposed, named DE-COMA. DE-COMA introduces individuals in a population as computational units to construct the policy network with operations such as mutation, crossover, and selection. The average return of DE-COMA is set as the fitness function, and the best individual of policy network will be chosen for the next generation. By maintaining better parameter sharing to enhance parameter diversity, multi-agent strategies will become more exploratory. To validate the effectiveness of DE-COMA, experiments were conducted in the StarCraft II environment with 2s_vs_1sc, 2s3z, 3m, and 8m battle scenarios. Experimental results demonstrate that DE-COMA significantly outperforms the traditional COMA and most other multi-agent reinforcement learning algorithms in terms of win rate and convergence speed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Military Defense Systems Analysis · Advanced Decision-Making Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Xi’an Technological University, Xi’an 710021, China
Jiangsu Automation Research Institute, Lianyungang 222061, China
Funding
National Foreign Expert Program of the Ministry of Science and Technology · Shaanxi Natural Science Basic Research Project · Science and Technology Program of Xi’an, China
References (30)
Show source-provided references
1. Towards Autonomous Multi-UAV Wireless Network: A Survey of Reinforcement Learning-Based Approaches10.1109/COMST.2023.3323344 ↗
2. Li, Y., Liu, I.J., Yuan, Y., Chen, D., Schwing, A., and Huang, J. (2019, January 22–26). Accelerating distributed reinforcement learning with in-switch computing. Proceedings of the 46th International Symposium on Computer Architecture, Phoenix, AZ, USA.10.1145/3307650.3322259 ↗
3. Reinforcement Learning Algorithms: A brief survey10.1016/j.eswa.2023.120495 ↗
4. Deep reinforcement learning: A survey10.1109/TNNLS.2022.3207346 ↗
5. A review of cooperative multi-agent deep reinforcement learning10.1007/s10489-022-04105-y ↗
6. Oliehoek, F.A., and Vlassis, N. (2007, January 14–18). Q-value functions for decentralized POMDPs. Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, Honolulu, HI, USA.10.1145/1329125.1329390 ↗
7. Omidshafiei, S., Pazis, J., Amato, C., How, J.P., and Vian, J. (2017, January 6–11). Deep decentralized multi-task multi-agent reinforcement learning under partial observability. Proceedings of the 34th International Conference on Machine Learning, PMLR, Sydney, Australia.
8. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
9. Han, D., Mulyana, B., Stankovic, V., and Cheng, S. (2023). A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation. Sensors, 23.10.3390/s23073762 ↗
10. Optimal payoff functions for members of collectives10.1142/S0219525901000188 ↗
11. A survey on evolutionary computation approaches to feature selection10.1109/TEVC.2015.2504420 ↗
12. Khadka, S., and Tumer, K. (2018, January 3–8). Evolution-guided policy gradient in reinforcement learning. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
13. Evolutionary Reinforcement Learning: A Survey10.34133/icomputing.0025 ↗
14. Wang, X., Zhang, Z., and Zhang, W. (2022). Model-based multi-agent reinforcement learning: Recent progress and prospects. arXiv.
15. Nash equilibrium seeking of general linear multi-agent systems in the cooperation–competition network10.1016/j.sysconle.2023.105510 ↗
16. Q-learning10.1007/BF00992698 ↗
17. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017). Multi-agent reinforcement learning in sequential social dilemmas. arXiv.
18. Genetic reinforcement learning for neurocontrol problems10.1023/A:1022674030396 ↗
19. Evolving neural networks through augmenting topologies10.1162/106365602320169811 ↗
20. Robots that can adapt like animals10.1038/nature14422 ↗
21. Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. (2017). Evolution strategies as a scalable alternative to reinforcement learning. arXiv.
22. Usunier, N., Synnaeve, G., Lin, Z., and Chintala, S. (2016). Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. arXiv.
23. Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., and Wang, J. (2017). Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv.
24. Jorge, E., Kågebäck, M., Johansson, F.D., and Gustavsson, E. (2016). Learning to play guess who? and inventing a grounded language as a consequence. arXiv.
25. Devlin, S., Yliniemi, L., Kudenko, D., and Tumer, K. (2014, January 5–9). Potential-based difference rewards for multiagent reinforcement learning. Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, Paris, France.
26. Recent advances in differential evolution—An updated survey10.1016/j.swevo.2016.01.004 ↗
27. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv.
28. Samvelyan, M., Rashid, T., De Witt, C.S., Farquhar, G., Nardelli, N., Rudner, T.G., Hung, C.M., Torr, P.H., Foerster, J., and Whiteson, S. (2019). The starcraft multi-agent challenge. arXiv.
29. Son, K., Kim, D., Kang, W.J., Hostallero, D.E., and Yi, Y. (2019, January 9–15). Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
30. Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. (2019, January 8–14). Maven: Multi-agent variational exploration. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.