Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences

2024-03-04 · Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanović, Adish Singla

In this paper, we take a step towards a deeper understanding of learning from human preferences by systematically comparing the paradigm of reinforcement learning from human feedback (RLHF) with the recently proposed paradigm of direct preference optimization (DPO). We focus our attention on the class of loglinear policy parametrization and linear reward functions. In order to compare the two paradigms, we first derive minimax statistical bounds on the suboptimality gap induced by both RLHF and DPO, assuming access to an oracle that exactly solves the optimization problems. We provide a detailed discussion on the relative comparison between the two paradigms, simultaneously taking into account the sample size, policy and reward class dimensions, and the regularization temperature. Moreover, we extend our analysis to the approximate optimization setting and derive exponentially decaying convergence rates for both RLHF and DPO. Next, we analyze the setting where the ground-truth reward is not realizable and find that, while RLHF incurs a constant additional error, DPO retains its asymptotically decaying gap by just tuning the temperature accordingly. Finally, we extend our comparison to the Markov decision process setting, where we generalize our results with exact optimization. To the best of our knowledge, we are the first to provide such a comparative analysis for RLHF and DPO.

📄 PDF Abstract BibTeX arXiv:2403.01857

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

DPO 설명 없음

Focus 설명 없음

CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization

2025-08-12 · Xinge Ye, Rui Wang, Yuchuan Wu, Victor Ma 외 arxiv

Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like rol…

Reinforcement Learning Mathematical Reasoning Code Generation

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

2023-09-30 · Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen 외

Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values…

reinforcement-learning Reinforcement Learning World Knowledge

A Technical Survey of Reinforcement Learning Techniques for Large Language Models

2025-07-05 · Saksham Sahai Srivastava, Vaneet Aggarwal arxiv

This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods. Additionall…

Reinforcement Learning Code Generation

Stepwise Alignment for Constrained Language Model Policy Optimization

2024-04-17 · Akifumi Wachi, Thien Q. Tran, Rei Sato, Takumi Tanabe 외

Safety and trustworthiness are indispensable requirements for real-world applications of AI systems using large language models (LLMs). This paper formulates human value alignment as an optimization problem of the langua…

Computational Efficiency Language Modeling Language Modelling model

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training

2023-01-18 · Philipp Altmann, Thomy Phan, Fabian Ritz, Thomas Gabor 외

We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store benef…

Deep Reinforcement Learning Imitation Learning