Balancing Depth for Robustness: A Study on Reincarnating Reinforcement Learning Models
Source abstract
Abstract
This paper investigates the impact of adaptive network depth selection on the robustness and performance of Regenerative Reinforcement Learning (RRL) models. RRL accelerates learning by reusing previously computed results. We propose a task-driven approach to dynamically configure network depth to explore the best architecture on Atari 2600 games. The experimental results show that a seven-layer network achieves the best balance between feature extraction, policy learning, and robustness, achieving an IQM score of 1.2 in a pool of 10 games. While deeper networks risk overfitting, shallower architectures struggle to cope with complex task dynamics. Our results highlight the key role of adaptive depth in enhancing the generalization, stability, and efficiency of RRL models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Adversarial Robustness in Machine Learning · Data Stream Mining Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, China
Funding
National Natural Science Foundation of China · State Key Laboratory of Simulation and Regulation of Water Cycle in River Basin, and China Institute of Water Resources and Hydropower Research, China
References (40)
Show source-provided references
1. Q-learning10.1007/BF00992698 ↗
2. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv.
3. Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
4. Transfer Learning for Reinforcement Learning Domains: A Survey
5. Finn, C., Abbeel, P., and Levine, S. (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv.
6. Fernández, F., and Veloso, M. Probabilistic policy reuse in a reinforcement learning agent. Proceedings of the AAMAS ’06 Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems 2006.10.1145/1160633.1160762 ↗
7. Human-level Control Through Deep Reinforcement Learning10.1038/nature14236 ↗
8. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
9. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
10. Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust Adversarial Reinforcement Learning. arXiv.
11. Boris, H., and Sellke, M. (2017). Approximating Continuous Functions by ReLU Nets of Minimal Width. arXiv.
12. Resnet with One-Neuron Hidden Layers Is A Universal Approximator
13. Telgarsky, M. (2015). Representation Benefits of Deep Feedforward Networks. arXiv.
14. Shin, Y. (2021). Effects of Depth, Width, and Initialization: A Convergence Analysis of Layer-Wise Training for Deep Linear Neural Networks. arXiv.10.1142/S0219530521500263 ↗
15. Cazenavette, G., Murdock, C., and Lucey, S. (2021). Architectural Adversarial Robustness: The Case for Deep Pursuit. arXiv.10.1109/CVPR46437.2021.00707 ↗
16. Su, D., Zhang, H., Chen, H., Yi, J., Chen, P.Y., and Gao, Y. (2018). Is Robustness the Cost of Accuracy?—A Comprehensive Study on the Robustness of 18 Deep Image Classification Models. arXiv.10.1007/978-3-030-01258-8_39 ↗
17. Nakkiran, P. (2019). Adversarial Robustness May Be at Odds with Simplicity. arXiv.
18. Wu, B., Chen, J., Cai, D., He, X., and Gu, Q. (2021). Do Wider Neural Networks Really Help Adversarial Robustness?. arXiv.
19. Huang, H., Wang, Y., Erfani, S., Gu, Q., Bailey, J., and Ma, X. (2021). Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks. arXiv.
20. Nguyen, T., Raghu, M., and Kornblith, S. (2021). Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. arXiv.
21. Tan, M., and Le, Q.V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv.
22. Zagoruyko, S., and Komodakis, N. (2016). Wide residual networks. arXiv.10.5244/C.30.87 ↗
23. Hacohen, G., Choshen, L., and Weinshall, D. (2020). Let’s agree to agree: Neural networks share classification order on real datasets. arXiv.
24. Hooker, S., Dauphin, Y., Courville, A., and Frome, A. (2019). Selective brain damage: Measuring the disparate impact of model pruning. arXiv.
25. Ba, J., and Caruana, R. (2014). Do deep nets really need to be deep?. arXiv.
26. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2015). Fitnets: Hints for thin deep nets. arXiv.
27. Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. arXiv.
28. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. (2017). Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. arXiv.
29. Morcos, A., Raghu, M., and Bengio, S. (2018). Insights on representational similarity in neural networks with canonical correlation. arXiv.
30. Gotmare, A., Keskar, N.S., Xiong, C., and Socher, R. (2018). A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv.
31. Neyshabur, B., Sedghi, H., and Zhang, C. (2020). What is being transferred in transfer learning?. arXiv.
32. Hermann, K., and Lampinen, A. (2020). What shapes feature representations? exploring datasets, architectures, and training. arXiv.
33. Maheswaranathan, N., Williams, A., Golub, M., Ganguli, S., and Sussillo, D. (2019). Universality and individuality in neural dynamics across large populations of recurrent networks. arXiv.
34. Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. (2019). Identifying and controlling important neurons in neural machine translation. arXiv.
35. Kudugunta, S.R., Bapna, A., Caswell, I., Arivazhagan, N., and Firat, O. (2019). Investigating multilingual nmt representations at scale. arXiv.10.18653/v1/D19-1167 ↗
36. Shi, J., Shea-Brown, E., and Buice, M. (2019). Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. arXiv.
37. Raghu, M., Zhang, C., Kleinberg, J., and Bengio, S. (2019). Transfusion: Understanding transfer learning for medical imaging. arXiv.
38. Ding, Z., Huang, Y., Yuan, H., and Dong, H. (2020). Introduction to Reinforcement Learning. Deep. Reinf. Learn., 47–123.10.1007/978-981-15-4095-0_2 ↗
39. Xiao, J., Fan, Y., Sun, R., and Luo, Z.Q. (2022). Adversarial Rademacher Complexity of Deep Neural Networks. arXiv.
40. Yin, D., Kannan, R., and Bartlett, P. (2018). Rademacher Complexity for Adversarially Robust Generalization. arXiv.