Noise-Aware Direct Preference Optimization for RLAIF
Source abstract
Abstract
Reinforcement Learning from Human Feedback (RLHF) produces powerful instruction-following models but relies on a preference-labeling process that is both costly and slow. An effective alternative, Reinforcement Learning from AI Feedback (RLAIF), uses large language models as teachers for relabeling; however, this introduces substantial label noise. In our setting, we found that AI teachers flipped approximately 50% of the original human preferences on the dataset, a condition that degrades the performance of standard direct preference optimization (DPO). We propose noise-robust DPO (nrDPO) and nrDPO-gated, two drop-in variants that make DPO resilient to noisy preferences. nrDPO reweights each pair by (i) a margin-confidence term from a frozen reference policy (base or SFT), (ii) a context-stability term that penalizes preferences that change under truncated histories, and (iii) a length correction to curb verbosity bias. nrDPO-gated further filters low-confidence pairs via a simple threshold on the reference margin. On a dataset with heavy synthetic noise (30% flips), nrDPO-gated improves the preference accuracy by +3.8% over vanilla DPO; in a realistic RLAIF setting, nrDPO-gated is the only configuration that recovers competitive alignment, reaching ≈60% on a 5k relabeled set (vs. ≈49–50% for vanilla DPO) and approaching RLHF baselines.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Control Systems Optimization · Advanced Multi-Objective Optimization Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Technology and Engineering, Kazakh-British Technical University, 050000 Almaty, Kazakhstan
AI Research Laboratory, Satbayev University, 050040 Almaty, Kazakhstan
Funding
Ministry of Science and Higher Education of the Republic of Kazakhstan
References (31)
Show source-provided references
1. Nguyen, N.T., Franczyk, B., Ludwig, A., Núñez, M., Treur, J., Vossen, G., and Kozierkiewicz, A. (2024). Enhancing Low-Resource NER via Knowledge Transfer from LLM. Computational Collective Intelligence: 16th International Conference, ICCCI 2024, Leipzig, Germany, 9–11 September 2024, Springer.10.1007/978-3-031-70816-9 ↗
2. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
3. Toleu, A., Tolegen, G., and Ualiyeva, I. (2025). Fine-Tuning Large Language Models for Kazakh Text Simplification. Appl. Sci., 15.10.20944/preprints202506.1947.v1 ↗
4. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems, Curran Associates, Inc.
5. Bouamor, H., Pino, J., and Bali, K. (2023, January 6–10). Learning Preference Model for LLMs via Automatic Preference Data Generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), Singapore.
6. Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., and Finn, C. (2023, January 10–16). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Proceedings of the Thirty-Seventh Conference on Neural Information Processing Systems, New Orleans, LA, USA.
7. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., and Henighan, T. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv.
8. A Survey of Preference-Based Reinforcement Learning Methods
9. Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2020). Fine-Tuning Language Models from Human Preferences. arXiv.
10. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv.
11. Learning to summarize with human feedback
12. Training language models to follow instructions with human feedback
13. Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. (2023). RRHF: Rank Responses to Align Language Models with Human Feedback without tears. arXiv.
14. Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. (2024). Preference Ranking Optimization for Human Alignment. arXiv.10.1609/aaai.v38i17.29865 ↗
15. Al-Onaizan, Y., Bansal, M., and Chen, Y. (2024, January 12–16). ORPO: Monolithic Preference Optimization without Reference Model. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.
16. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., and McKinnon, C. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv.
17. Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. (2024). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv.
18. Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Ramé, A., Mesnard, T., Zhao, Y., and Piot, B. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv.
19. Yuan, W., Pang, R.Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. (2025). Self-Rewarding Language Models. arXiv.
20. Liu, A., Bai, H., Lu, Z., Kong, X., Wang, S., Shan, J., Cao, M., and Wen, L. (2024). Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation. arXiv.10.18653/v1/2024.acl-long.523 ↗
21. Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. (2024). RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. arXiv.
22. Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., and Xing, E.P. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv.
23. Al-Onaizan, Y., Bansal, M., and Chen, Y. (2024, January 12–16). Humans or LLMs as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.
24. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T.B. (2025). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv.
25. Sharma, A., Keh, S., Mitchell, E., Finn, C., Arora, K., and Kollar, T. (2024). A Critical Evaluation of AI Feedback for Aligning Large Language Models. arXiv.
26. Chowdhury, S.R., Kini, A., and Natarajan, N. (2024). Provably Robust DPO: Aligning Language Models with Noisy Feedback. arXiv.
27. Liang, X., Chen, C., Qiu, S., Wang, J., Wu, Y., Fu, Z., Shi, Z., Wu, F., and Ye, J. (2024). ROPO: Robust Preference Optimization for Large Language Models. arXiv.
28. Bukharin, A., Hong, I., Jiang, H., Li, Z., Zhang, Q., Zhang, Z., and Zhao, T. (2024). Robust Reinforcement Learning from Corrupted Human Feedback. arXiv.
29. Yan, Y., Lou, X., Li, J., Zhang, Y., Xie, J., Yu, C., Wang, Y., Yan, D., and Shen, Y. (2024). Reward-Robust RLHF in LLMs. arXiv.
30. Zhang, C., Shen, W., Zhao, L., Zhang, X., Xu, X., Dou, W., and Bian, J. (2025). Policy Filtration for RLHF to Mitigate Noise in Reward Models. arXiv.
31. Addula, S.R., Shukla, A., Chaurasia, V.R., Kumar, A., Akula, A.P., and Harini, R. (2024, January 20–21). Dynamic Load Balancing in Cloud Computing using Hybrid Kookaburra–Pelican Optimization Algorithms. Proceedings of the International Conference on Augmented Reality, Intelligent Systems, and Industrial Automation (ARIIA), Manipal, India.10.1109/ARIIA63345.2024.11051893 ↗