Reinforcement Learning in Game Industry—Review, Prospects and Challenges
Source abstract
Abstract
This article focuses on the recent advances in the field of reinforcement learning (RL) as well as the present state–of–the–art applications in games. First, we give a general panorama of RL while at the same time we underline the way that it has progressed to the current degree of application. Moreover, we conduct a keyword analysis of the literature on deep learning (DL) and reinforcement learning in order to analyze to what extent the scientific study is based on games such as ATARI, Chess, and Go. Finally, we explored a range of public data to create a unified framework and trends for the present and future of this sector (RL in games). Our work led us to conclude that deep RL accounted for roughly 25.1% of the DL literature, and a sizable amount of this literature focuses on RL applications in the game domain, indicating the road for newer and more sophisticated algorithms capable of outperforming human performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Artificial Intelligence in Games · Sports Analytics and Performance
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
MLV Research Group, Department of Computer Science, International Hellenic University, 65404 Kavala, Greece
References (64)
Show source-provided references
1. Broy, M. (2011). Software Pioneers, Springer.
2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.F. (2009, January 20–25). ImageNet: A large–scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
3. Duy, T., Sato, Y., and Inoguchi, Y. (2010, January 19–23). Performance evaluation of a green scheduling algorithm for energy savings in cloud computing. Proceedings of the 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Ph.D. Forum (IPDPSW), Atlanta, GA, USA.10.1109/IPDPSW.2010.5470908 ↗
4. Prevost, J., Nagothu, K., Kelley, B., and Jamshidi, M. (2011, January 27–30). Prediction of cloud data center networks loads using stochastic and neural models. Proceedings of the 2011 6th International Conference on System of Systems Engineering, Albuquerque, NM, USA.10.1109/SYSOSE.2011.5966610 ↗
5. Machine learning based resource allocation of cloud computing in auction
6. Yang, R., Ouyang, X., Chen, Y., Townend, P., and Xu, J. (2018, January 26–29). Intelligent resource scheduling at scale: A machine learning perspective. Proceedings of the 2018 IEEE Symposium on Service–Oriented System Engineering, Bamberg, Germany.10.1109/SOSE.2018.00025 ↗
7. Empirical prediction models for adaptive resource provisioning in the cloud10.1016/j.future.2011.05.027 ↗
8. Bellemare, M.G., Naddaf, Y., Veness, J., and Bowling, M. (2012). The Arcade Learning Environment: An Evaluation Platform for General Agents. arXiv.10.1613/jair.3912 ↗
9. A Neuroevolution Approach to General Atari Game Playing10.1109/TCIAIG.2013.2294713 ↗
10. Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (2017). A Brief Survey of Deep Reinforcement Learning. arXiv.10.1109/MSP.2017.2743240 ↗
11. Fujimoto, S., van Hoof, H., and Meger, D. (2018, January 10–15). Addressing Function Approximation Error in Actor–Critic Methods. Proceedings of the 2018 International Conference on Machine Learning, Stockholm, Sweden. arXiv:1802.09477.
12. Anschel, O., Baram, N., and Shimkin, N. (2017, January 6–11). Averaged–DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia.
13. Hasselt, H., Guez, A., and Silver, D. (2016, January 12–17). Deep reinforcement learning with double q–learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.
14. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016, January 19–24). Dueling network architectures for deep reinforcement learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
15. Schulman, J., Levine, S., Moritz, P., Jordan, M.I., and Abbeel, P. (2015, January 6–11). Trust Region Policy Optimization. Proceedings of the 2015 International Conference on Machine Learning, Lille, France.
16. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
17. Hausknecht, M., and Stone, P. (2015, January 12–14). Deep recurrent q–learning for partially observable mdps. Proceedings of the 2015 AAAI Fall Symposium Series, Arlington, VA, USA.
18. Variants of Bellman equation on reinforcement learning problems
19. Human–level control through deep reinforcement learning10.1038/nature14236 ↗
20. A Systematic Review on Reinforcement Learning–Based Robotics Within the Last Decade10.1109/ACCESS.2020.3027152 ↗
21. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. A Bradford Book.
22. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized experience replay. arXiv.
23. Lin, L.J. (1993). Reinforcement Learning for Robots Using Neural Networks. [Ph.D. Thesis, Carnegie Mellon University].
24. Fortunato, M., Azar, M., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., and Pietquin, O. (2017). Noisy networks for exploration. arXiv.
25. Osband, I., Blundell, C., Pritzel, A., and Roy, B. (2023, January 03). Deep Exploration via Bootstrapped Dqn. Advances in Neural Information Processing Systems. Available online: https://proceedings.neurips.cc/paper/2016/file/8d8818c8e140c64c743113f563cf750f-Paper.pdf.
26. Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., and Osband, I. (2018, January 2–7). Deep q–learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11757 ↗
27. Andrew, A. (1999). Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto, Adaptive Computation and Machine Learning Series, MIT Press.
28. An analysis of temporal–difference learning with function approximation10.1109/9.580874 ↗
29. McDermott, J., Castelli, M., Sekanina, L., Haasdijk, E., and García–Sánchez, P. (2017). Proceedings of the Genetic Programming, Springer International Publishing.10.1007/978-3-319-55696-3 ↗
30. Wilson, D.G., Cussat–Blanc, S., Luga, H., and Miller, J.F. (2018). Proceedings of the Genetic and Evolutionary Computation Conference, Kyoto, Japan, 15–19 July 2018, Association for Computing Machinery.
31. Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., and García–Sánchez, P. (2018). Proceedings of the Genetic Programming, Springer International Publishing.10.1007/978-3-319-77553-1 ↗
32. Smith, R.J., and Heywood, M.I. (2019). Proceedings of the Genetic and Evolutionary Computation Conference, Prague, Czech Republic, 13–17 July 2019, Association for Computing Machinery.
33. Zhao, D., Wang, H., Shao, K., and Zhu, Y. (2016, January 6–9). Deep reinforcement learning with experience replay based on sarsa. Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence, Athens, Greece.10.1109/SSCI.2016.7849837 ↗
34. Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv.
35. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21–26). Deterministic policy gradient algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China.
36. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead10.1038/s42256-019-0048-x ↗
37. Melnik, A., Fleer, S., Schilling, M., and Ritter, H. (2019). Modularization of end–to–end learning: Case study in arcade games. arXiv.
38. Polvara, R., Patacchiola, M., Sharma, S., Wan, J., Manning, A., Sutton, R., and Cangelosi, A. (2017). Autonomous quadrotor landing using deep reinforcement learning. arXiv.
39. Pan, X., You, Y., Wang, Z., and Lu, C. (2017). Virtual to real reinforcement learning for autonomous driving. arXiv.10.5244/C.31.11 ↗
40. Loiacono, D., Cardamone, L., and Lanzi, P. (2013). Simulated car racing championship: Competition software manual. arXiv.
41. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv.
42. Synnaeve, G., Nardelli, N., Auvolat, A., Chintala, S., Lacroix, T., Lin, Z., Richoux, F., and Usunier, N. (2016). Torchcraft: A library for machine learning research on real–time strategy games. arXiv.
43. Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., and Wang, J. (2017). Multiagent bidirectionally–coordinated nets: Emergence of humanlevel coordination in learning to play starcraft combat games. arXiv.
44. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018, January 2–7). Counterfactual multi–agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11794 ↗
45. The ins and outs of fmri signals10.1038/nn1007-1230 ↗
46. Oh, J., Singh, S., Lee, H., and Kohli, P. (2017, January 6–11). Zero–shot task generalization with multi–task deep reinforcement learning. Proceedings of the International Conference on Machine Learning, Sydney, NSW, Australia.
47. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2017). OpenAI Gym, 2016. arXiv.
48. Xiong, Y., Chen, H., Zhao, M., and An, B. (2018, January 2–7). Hogrider: Champion agent of microsoft malmo collaborative ai challenge. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11581 ↗
49. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., and Graepel, T. (2017). Mastering chess and shogi by self–play with a general reinforcement learning algorithm. arXiv.
50. David, O., Netanyahu, N., and Wolf, L. (2016). Proceedings of the International Conference on Artificial Neural Networks, Barcelona, Spain, 6–9 September 2016, Springer.
51. A general reinforcement learning algorithm that masters chess, shogi, and go through self–play10.1126/science.aar6404 ↗
52. Justesen, N., Torrado, R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S. (2018). Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv.
53. Choudhary, A. (2022, October 21). A Hands-On Introduction to Deep Q-Learning Using OpenAI Gym in Python. Anal Vidhya. Available online: https://www.analyticsvidhya.com/blog/2019/04/introduction-deep-q-learning-python/.
54. Arthur guez, an cjm10.1038/nature16961 ↗
55. OpenAI, Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., and Hashme, S. (2019). Dota 2 with large scale deep reinforcement learning. arXiv.
56. Unifying count–based exploration and intrinsic motivation
57. Long short–term memory10.1162/neco.1997.9.8.1735 ↗
58. Such, F., Madhavan, V., Conti, E., Lehman, J., Stanley, K., and Clune, J. (2017). Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv.
59. Vinyals, O., Fortunato, M., and Jaitly, N. (2015). Pointer networks. arXiv.
60. Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y.W. (2018, January 15–19). Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. Proceedings of the International Conference on Machine Learning, Stockholmsmässan, Sweden.
61. Ammanabrolu, P., and Riedl, M.O. (2018). Playing Text–Adventure Games with Graph–Based Deep Reinforcement Learning. arXiv.10.18653/v1/N19-1358 ↗
62. Adolphs, L., and Hofmann, T. (2019, January 27–28). LeDeepChef: Deep Reinforcement Learning Agent for Families of Text–Based Games. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
63. Brown, N., Bakhtin, A., Lerer, A., and Gong, Q. (2020). Combining Deep Reinforcement Learning and Search for Imperfect–Information Games. arXiv.
64. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning