Efficient Classification-Based Constraints for Offline Reinforcement Learning
Source abstract
Abstract
Existing distribution-based constraints in offline reinforcement learning, such as bootstrapping error accumulation reduction (BEAR), can prevent policy deviation but often ignore action quality, leading to O(n2) complexity and limited interpretability. This study introduces a classification-based approach that employs pairwise action quality comparison to replace complex distributional constraints. A binary classifier learns the relative quality of two actions in the same state by comparing their Q-values, prioritizing value-aware selection while maintaining conservative behavior. Experiments on benchmark environments demonstrate that the proposed method consistently improves upon BEAR, achieving 3× on average and up to 5× in some environments. The algorithm reduces computational complexity from O(n2) to O(n) while providing intuitive monitoring through classifier accuracy. These results indicate that efficient quality-based comparisons can serve as a practical and efficient alternative to computationally expensive distributional constraints in offline reinforcement learning, yielding practical gains in both performance and scalability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reinforcement Learning in Robotics · Evolutionary Algorithms and Applications · Advanced Multi-Objective Optimization Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Bright College (College of Liberal Arts and Interdisciplinary Studies), Hankyong National University, 327, Jungang-ro, Anseong-si 17579, Gyeonggi-do, Republic of Korea
References (43)
Show source-provided references
1. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].
2. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
3. Li, Y. (2017). Deep Reinforcement Learning: An Overview. arXiv.
4. Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv.
5. Batch reinforcement learning10.1007/978-3-642-27645-3_2 ↗
6. Kumar, A., Fu, J., Tucker, G., and Levine, S. (2019, January 8–14). Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
7. Fujimoto, S., Meger, D., and Precup, D. (2019, January 9–15). Off-Policy Deep Reinforcement Learning without Exploration. Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA.
8. A Kernel Two-Sample Test
9. Kostrikov, I., Nair, A., and Levine, S. (2022, January 25–29). Offline Reinforcement Learning with Implicit Q-Learning. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event.
10. Gülçehre, Ç., Srinivasan, S., Sygnowski, J., Ostrovski, G., Farajtabar, M., Hoffman, M., Pascanu, R., and Doucet, A. (2022). An Empirical Study of Implicit Regularization in Deep Offline RL. arXiv.
11. Wu, Y., Tucker, G., and Nachum, O. (2020, January 26–30). Behavior Regularized Offline Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
12. Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017, January 4–9). Deep Reinforcement Learning from Human Preferences. Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
13. O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. (2018, January 10–15). The Uncertainty Bellman Equation and Exploration. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden.
14. Luo, Q.-W., Xie, M.-K., Wang, Y.-W., and Huang, S.-J. (2025). Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL. arXiv.
15. Duan, X., He, Y., Tajwar, F., Salakhutdinov, R., Kolter, J.Z., and Schneider, J. (2025). Accelerating Diffusion Models in Offline RL via Reward-Aware Consistency Trajectory Distillation. arXiv.
16. Laroche, R., Trichelair, P., and Tachet des Combes, R. (2019, January 9–15). Safe Policy Improvement with Baseline Bootstrapping. Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA.
17. Agarwal, R., Schuurmans, D., and Norouzi, M. (2020, January 13–18). An Optimistic Perspective on Offline Reinforcement Learning. Proceedings of the 37th International Conference on Machine Learning (ICML), Virtual Event.
18. Kim, C. (2025). Classification-Based Q-Value Estimation for Continuous Actor-Critic Reinforcement Learning. Symmetry, 17.10.3390/sym17050638 ↗
19. Ishfaq, H., Nguyen-Tang, T., Feng, S., Arora, R., Wang, M., Yin, M., and Precup, D. (2024, January 10–16). Offline Multitask Representation Learning for Reinforcement Learning. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
20. Lien, Y.-H., Hsieh, P.-C., Li, T.-M., and Wang, Y.-S. (2024, January 21–27). Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning. Proceedings of the 41st International Conference on Machine Learning (ICML), Vienna, Austria.
21. Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020, January 6–12). Conservative Q-Learning for Offline Reinforcement Learning. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), Virtual Event.
22. Nair, A., Gupta, A., Dalal, M., and Levine, S. (2020). AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv.
23. Peng, X.B., Kumar, A., Zhang, G., and Levine, S. (2019). Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv.
24. Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. (2020, January 6–12). MOPO: Model-Based Offline Policy Optimization. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), Virtual Event.
25. Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. (2021, January 18–24). COMBO: Conservative Offline Model-Based Policy Optimization. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual Event.
26. Burges, C.J.C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. (2005, January 7–11). Learning to Rank Using Gradient Descent. Proceedings of the 22nd International Conference on Machine Learning (ICML), Bonn, Germany.10.1145/1102351.1102363 ↗
27. Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., and Li, H. (2007, January 20–24). Learning to Rank: From Pairwise Approach to Listwise Approach. Proceedings of the 24th International Conference on Machine Learning (ICML), Corvallis, OR, USA.10.1145/1273496.1273513 ↗
28. Singh, A., Yu, A., Yang, J., Zhang, J., Kumar, A., and Levine, S. (2021, January 8–11). COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning. Proceedings of the 5th Conference on Robot Learning (CoRL), London, UK.
29. Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., and de Freitas, N. (2016, January 19–24). Dueling Network Architectures for Deep Reinforcement Learning. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA.
30. Bellemare, M.G., Dabney, W., and Munos, R. (2017, January 6–11). A Distributional Perspective on Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, Australia.
31. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021, January 6–14). Decision Transformer: Reinforcement Learning via Sequence Modeling. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS), Virtual Event.
32. Choi, H., Jung, S., Ahn, H., and Moon, T. (2024, January 21–27). Listwise Reward Estimation for Offline Preference-Based Reinforcement Learning. Proceedings of the 41st International Conference on Machine Learning (ICML), Vienna, Austria.
33. Lei, K., He, Z., Lu, C., Hu, K., Gao, Y., and Xu, H. (2024, January 7–11). Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization. Proceedings of the International Conference on Learning Representations (ICLR 2024), Vienna, Austria. Available online: https://openreview.net/forum?id=tbFBh3LMKi.
34. Shin, Y., Kim, J., Jung, W., Hong, S., Yoon, D., Jang, Y., Kim, G., Chae, J., Sung, Y., and Lee, K. (2025). Online Pre-Training for Offline-to-Online Reinforcement Learning. arXiv.
35. Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., and Zhang, Y.-Q. (2023). PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning. arXiv.
36. Building Explainable AI for Reinforcement Learning-Based Debt Collection Recommender Systems
37. Imagination-Limited Q-Learning for Safe Offline Reinforcement Learning
38. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.
39. Farama Foundation (2025, August 01). Gymnasium: A Standard API for Reinforcement Learning. Available online: https://gymnasium.farama.org/.
40. Farama Foundation (2025, August 01). Pendulum-v1. Available online: https://gymnasium.farama.org/environments/classic_control/pendulum/.
41. Farama Foundation (2025, August 01). MountainCarContinuous-v0. Available online: https://gymnasium.farama.org/environments/classic_control/mountain_car_continuous/.
42. Todorov, E., Erez, T., and Tassa, Y. (2012, January 7–12). MuJoCo: A Physics Engine for Model-based Control. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura-Algarve, Portugal.10.1109/IROS.2012.6386109 ↗
43. The Role of Occam’s Razor in Knowledge Discovery10.1023/A:1009868929893 ↗