Balanced Domain Randomization for Safe Reinforcement Learning
Source abstract
Abstract
Reinforcement Learning (RL) has enabled autonomous agents to achieve superhuman performance in diverse domains, including games, navigation, and robotic control. Despite these successes, RL agents often struggle with overfitting to specific training environments, which can hinder their adaptability to new or rare scenarios. Domain randomization is a widely used technique designed to enhance policy robustness by training models in a variety of randomized contexts. In traditional domain randomization, however, models tend to prioritize learning from more common domains, often neglecting rare ones. To address this imbalance, we propose Balanced Domain Randomization (BDR) that balances the learning focus based on the rarity of contexts. We assess context rarity in the embedding space using statistical analysis of context vectors and adjust the loss weights for each transition based on this rarity. This ensures that the agent dedicates adequate attention to rare domains. Our experimental results show that BDR efficiently enhances worst-case performance, significantly improving the robustness of RL-based robotic controllers across diverse conditions. This study provides a robust framework for RL agents, reducing risks in uncommon scenarios and ensuring reliable performance in varied environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Adaptive Dynamic Programming Control · Robot Manipulation and Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
INEEJI Corporation, Seongnam 13558, Republic of Korea
Funding
Unmanned Swarm CPS Research Laboratory Program of Defense Acquisition Program Administration and Agency for Defense Development
References (50)
Show source-provided references
1. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
2. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv.
3. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection10.1177/0278364917710318 ↗
4. Mastering the game of go without human knowledge10.1038/nature24270 ↗
5. Mastering atari, go, chess and shogi by planning with a learned model10.1038/s41586-020-03051-4 ↗
6. Grandmaster level in StarCraft II using multi-agent reinforcement learning10.1038/s41586-019-1724-z ↗
7. Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J., and Whiteson, S. (2024, January 10–15). Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
8. Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore.
9. Learning dexterous in-hand manipulation10.1177/0278364919887447 ↗
10. Patel, U., Kumar, N.K.S., Sathyamoorthy, A.J., and Manocha, D. (June, January 30). Dwa-rl: Dynamically feasible deep reinforcement learning policy for robot navigation among mobile obstacles. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.
11. Deep reinforcement learning based mobile robot navigation: A review10.26599/TST.2021.9010012 ↗
12. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017, January 24–28). Domain randomization for transferring deep neural networks from simulation to the real world. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202133 ↗
13. Yu, W., Tan, J., Liu, C.K., and Turk, G. (2017, January 12–16). Preparing for the unknown: Learning a universal policy with online system identification. Proceedings of the Robotics: Science and Systems (RSS), Cambridge, MA, USA.10.15607/RSS.2017.XIII.048 ↗
14. Peng, X.B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018, January 21–25). Sim-to-real transfer of robotic control with dynamics randomization. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460528 ↗
15. Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., and Vanhoucke, V. (2018, January 26–30). Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. Proceedings of the Robotics: Science and Systems (RSS), Pittsburgh, PA, USA.10.15607/RSS.2018.XIV.010 ↗
16. Chen, X., Hu, J., Jin, C., Li, L., and Wang, L. (2021, January 4). Understanding Domain Randomization for Sim-to-real Transfer. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
17. Luo, F.M., Jiang, S., Yu, Y., Zhang, Z., and Zhang, Y.F. (March, January 22). Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Virtual.
18. Hallak, A., Di Castro, D., and Mannor, S. (2015). Contextual markov decision processes. arXiv.
19. Benjamins, C., Eimer, T., Schubert, F., Mohan, A., Döhler, S., Biedenkapp, A., Rosenhahn, B., Hutter, F., and Lindauer, M. (2022). Contextualize Me—The Case for Context in Reinforcement Learning. arXiv.
20. Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. (2019, January 10–15). Efficient off-policy meta-reinforcement learning via probabilistic context variables. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
21. Zhou, W., Pinto, L., and Gupta, A. (2019, January 6–9). Environment probing interaction policies. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA.
22. Lee, K., Seo, Y., Lee, S., Lee, H., and Shin, J. (2020, January 12–18). Context-aware dynamics model for generalization in model-based reinforcement learning. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
23. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning
24. Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., and Song, D. (2018). Assessing generalization in deep reinforcement learning. arXiv.
25. Zhao, W., Queralta, J.P., and Westerlund, T. (2020, January 1–4). Sim-to-real transfer in deep reinforcement learning for robotics: A survey. Proceedings of the Symposium Series on Computational Intelligence, Canberra, Australia.10.1109/SSCI47803.2020.9308468 ↗
26. A survey of zero-shot generalisation in deep reinforcement learning10.1613/jair.1.14174 ↗
27. SMOTE: Synthetic minority over-sampling technique10.1613/jair.953 ↗
28. Drummond, C., and Holte, R.C. (2003, January 21). C4. 5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling. Proceedings of the Workshop on Learning from Imbalanced Datasets II, Washington, DC, USA.
29. A multiple resampling method for learning from imbalanced data sets10.1111/j.0824-7935.2004.t01-1-00228.x ↗
30. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.324 ↗
31. Ren, M., Zeng, W., Yang, B., and Urtasun, R. (2018, January 25–31). Learning to reweight examples for robust deep learning. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
32. Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. (2019, January 8–14). Learning imbalanced datasets with label-distribution-aware margin loss. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
33. Cui, Y., Jia, M., Lin, T.Y., Song, Y., and Belongie, S. (2019, January 15–20). Class-balanced loss based on effective number of samples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00949 ↗
34. Alshammari, S., Wang, Y.X., Ramanan, D., and Kong, S. (2022, January 18–24). Long-tailed recognition via weight balancing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00677 ↗
35. Liu, S., Garrepalli, R., Dietterich, T., Fern, A., and Hendrycks, D. (2018, January 25–31). Open category detection with PAC guarantees. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
36. Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M. (2019, January 15–20). Feature transfer learning for face recognition with under-represented data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00585 ↗
37. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009, January 14–18). Curriculum learning. Proceedings of the International Conference on Machine Learning (ICML), Montreal, QC, Canada.10.1145/1553374.1553380 ↗
38. Curriculum learning for reinforcement learning domains: A framework and survey
39. Kim, S., Lee, K., and Choi, J. (2023, January 23–29). Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skills. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA.
40. Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., and Yoo, J. (2020, January 12–18). Reliable fidelity and diversity metrics for generative models. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
41. Han, J., Choi, H., Choi, Y., Kim, J., Ha, J.W., and Choi, J. (2023, January 1–5). Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
42. The optimal control of partially observable Markov processes over a finite horizon10.1287/opre.21.5.1071 ↗
43. Planning and acting in partially observable stochastic domains10.1016/S0004-3702(98)00023-X ↗
44. On the generalized distance in statistics
45. Spacetime discounted value of network connectivity10.1142/S0219525918500182 ↗
46. Towers, M., Kwiatkowski, A., Terry, J.K., Balis, J.U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., KG, A., and Krimmel, M. (2024). Gymnasium: A standard interface for reinforcement learning environments. arXiv.
47. dm_control: Software and tasks for continuous control10.1016/j.simpa.2020.100022 ↗
48. Fujimoto, S., Hoof, H., and Meger, D. (2018, January 25–31). Addressing function approximation error in actor-critic methods. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
49. Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., and Ribas, R. (2019). Solving rubik’s cube with a robot hand. arXiv.
50. Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. (2023, January 10–16). Safety gymnasium: A unified safe reinforcement learning benchmark. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.