Policy-labeled Preference Learning: Is Preference Enough for RLHF?

2025-05-06 · Taehyun Cho, Seokhun Ju, Seungyub Han, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee

To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. Inspired by Direct Preference Optimization framework which directly learns optimal policy without explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret, which reflects behavior policy information. We also provide a contrastive KL regularization, derived from regret-based principles, to enhance RLHF in sequential decision making. Experiments in high-dimensional continuous control tasks demonstrate PPL's significant improvements in offline RLHF performance and its effectiveness in online settings.

📄 PDF Abstract BibTeX arXiv:2505.06273

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control Decision Making reinforcement-learning Reinforcement LearningSequential Decision Making

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning

2024-07-02 · Yifang Chen, Shuohang Wang, ZiYi Yang, Hiteshi Sharma 외

Reinforcement learning with human feedback (RLHF), as a widely adopted approach in current large language model pipelines, is \textit{bottlenecked by the size of human preference data}. While traditional methods rely on …

Active Learning Language Modelling Large Language Model MMLU

COPR: Continual Learning Human Preference through Optimal Policy Regularization

2023-10-24 · Han Zhang, Lin Gui, Yuanzhao Zhai, Hui Wang 외

The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the cur…

Continual Learning reinforcement-learning Reinforcement Learning

Preference as Reward, Maximum Preference Optimization with Importance Sampling

2023-12-27 · Zaifan Jiang, Xing Huang, Chao Wei

Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a rew…

Accelerated Preference Optimization for Large Language Model Alignment

2024-10-08 · Jiafan He, Huizhuo Yuan, Quanquan Gu

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO), one of the most popular approaches, …

Language Modeling Language Modelling Large Language Model

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

2024-05-29 · Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai 외

Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline R…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Text Summarization