World-Models for Bitrate Streaming
Source abstract
Abstract
Adaptive bitrate (ABR) algorithms optimize the quality of streaming experiences for users in client-side video players, especially in unreliable or slow mobile networks. Several rule-based heuristic algorithms can achieve stable performance, but they sometimes fail to properly adapt to changing network conditions. Fluctuating bandwidth may cause algorithms to default to behavior that creates a negative experience for the user. ABR algorithms can be generated with reinforcement learning, a decision-making paradigm in which an agent learns to make optimal choices through interactions with an environment. Training reinforcement learning algorithms for bitrate streaming requires building a simulator for an agent to experience interactions quickly; training an agent in the real environment is infeasible due to the long step times in real environments. This project explores using supervised learning to construct a world-model, or a learned simulator, from recorded interactions. A reinforcement learning agent that is trained inside of the learned model, rather than a simulator, can outperform rule-based heuristics. Furthermore, agents that are trained inside the learned world-model can outperform model-free agents in low sample regimes. This work highlights the potential for world-models to quickly learn simulators, and to be used for generating optimal policies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image and Video Quality Assessment · Peer-to-Peer Network Technologies · Advanced Bandit Algorithms Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Laboratory, University of Cambridge, Cambridge CB3 0FD, UK
References (35)
Show source-provided references
1. Mao, H., Netravali, R., and Alizadeh, M. (2017, January 21–25). Neural Adaptive Video Streaming with Pensieve. Proceedings of the Conference of the ACM Special Interest Group on Data Communication, Los Angeles, CA, USA.10.1145/3098822.3098843 ↗
2. Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (2017). A Brief Survey of Deep Reinforcement Learning. arXiv.10.1109/MSP.2017.2743240 ↗
3. Reinforcement Learning: A Survey10.1613/jair.301 ↗
4. Schmidhuber, J. (2015). On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models. arXiv.
5. Ha, D., and Schmidhuber, J. (2018). Recurrent World Models Facilitate Policy Evolution. Advances in Neural Information Processing Systems 31, Curran Associates, Inc.. Available online: https://worldmodels.github.io.
6. Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R.H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., and Levine, S. (2019). Model-Based Reinforcement Learning for Atari. arXiv.
7. Mao, H., Negi, P., Narayan, A., Wang, H., Yang, J., Wang, H., Marcus, R., Shirkoohi, M.K., He, S., and Nathan, V. (2019). Park: An Open Platform for Learning-Augmented Computer Systems. Advances in Neural Information Processing Systems, Curran Associates Inc.
8. (2016, August 18). Akamai. dash.js. Available online: https://github.com/Dash-Industry-Forum/dash.js/.
9. Huang, T.Y., Handigol, N., Heller, B., McKeown, N., and Johari, R. (2012, January 14–16). Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. Proceedings of the 2012 Internet Measurement Conference, Boston, MA, USA.10.1145/2398776.2398800 ↗
10. Bemporad, A., and Morari, M. (1999). Robust Model Predictive Control: A Survey. Robustness in Identification and Control, Springer.10.1007/BFb0109870 ↗
11. Spiteri, K., Urgaonkar, R., and Sitaraman, R.K. (2015, January 10–15). BOLA: Near-Optimal Bitrate Adaptation for Online Videos. Proceedings of the IEEE INFOCOM 2016—The 35th Annual IEEE International Conference on Computer Communications, San Francisco, CA, USA.10.1109/INFOCOM.2016.7524428 ↗
12. Huang, T.Y., Johari, R., McKeown, N., Trunnell, M., and Watson, M. (2014, January 17–22). A Buffer-Based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. Proceedings of the 2014 ACM Conference on SIGCOMM, Chicago, IL, USA.10.1145/2619239.2626296 ↗
13. Chebotar, Y., Hausman, K., Zhang, M., Sukhatme, G., Schaal, S., and Levine, S. (2017, January 6–11). Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
14. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv.
15. Mastering the game of Go with deep neural networks and tree search10.1038/nature16961 ↗
16. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19–24). Asynchronous Methods for Deep Reinforcement Learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
17. Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A.S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). StarCraft II: A New Challenge for Reinforcement Learning. arXiv.
18. Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., and Farhadi, A. (June, January 29). Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989381 ↗
19. Konda, V.R., and Tsitsiklis, J.N. (2000). Actor-Critic Algorithms. Advances in Neural Information Processing Systems, MIT Press.
20. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018, January 2–7). Rainbow: Combining Improvements in Deep Reinforcement Learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11796 ↗
21. Dulac-Arnold, G., Mankowitz, D., and Hester, T. (2019). Challenges of Real-World Reinforcement Learning. arXiv.
22. Haj-Ali, A., Ahmed, N.K., Willke, T., Gonzalez, J., Asanovic, K., and Stoica, I. (2019). Deep Reinforcement Learning in System Optimization. arXiv.
23. Buesing, L., Weber, T., Racaniere, S., Eslami, S., Rezende, D., Reichert, D.P., Viola, F., Besse, F., Gregor, K., and Hassabis, D. (2018). Learning and Querying Fast Generative Models for Reinforcement Learning. arXiv.
24. Deep Learning in Neural Networks: An Overview10.1016/j.neunet.2014.09.003 ↗
25. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
26. Graves, A. (2013). Generating Sequences With Recurrent Neural Networks. arXiv.10.1007/978-3-642-24797-2_3 ↗
27. Bishop, C.M. (2020, August 18). Mixture Density Networks. Available online: http://publications.aston.ac.uk/id/eprint/373/.
28. The Arcade Learning Environment: An Evaluation Platform for General Agents10.1613/jair.3912 ↗
29. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016, January 19–24). Benchmarking Deep Reinforcement Learning for Continuous Control. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
30. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.
31. Riiser, H., Vigmostad, P., Griwodz, C., and Halvorsen, P. (March, January 28). Commute Path Bandwidth Traces from 3G Networks: Analysis and Applications. Proceedings of the 4th ACM Multimedia Systems Conference, Oslo, Norway.
32. Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., and Nichol, A. (2018, August 18). Stable Baselines. Available online: https://github.com/hill-a/stable-baselines.
33. Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019, January 4–8). Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, AK, USA.10.1145/3292500.3330701 ↗
34. Bergstra, J.S., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems, Curran Associates, Inc.
35. Doshi-Velez, F., and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv.