Reward-Robust RLHF in LLMs

2024-09-18 · Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen

As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our approach introduces a novel optimization objective that carefully balances performance and robustness by incorporating Bayesian Reward Model Ensembles (BRME) to model the uncertainty set of reward functions. This allows the framework to integrate both nominal performance and minimum reward signals, ensuring more stable learning even with imperfect RMs. Empirical results demonstrate that our framework consistently outperforms baselines across diverse benchmarks, showing improved accuracy and long-term stability. We also provide a theoretical analysis, demonstrating that reward-robust RLHF approaches the stability of constant reward settings, which proves to be acceptable even in a stochastic-case analysis. Together, these contributions highlight the framework potential to enhance both the performance and stability of LLM alignment.

📄 PDF Abstract BibTeX arXiv:2409.15360

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

2024-08-30 · Jiayi Zhou, Jiaming Ji, Juntao Dai, Yaodong Yang

Aligning the behavior of Large language models (LLMs) with human intentions and values remains a critical challenge. Reinforcement learning from human feedback (RLHF) aligns LLMs by training a reward model (RM) on human …

Text Summarization

Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

2024-12-08 · Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du 외

This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potenti…

Diversity

Reward Difference Optimization For Sample Reweighting In Offline RLHF

2024-08-18 · Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan 외

With the rapid advances in Large Language Models (LLMs), aligning LLMs with human preferences become increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated …

Generative RLHF-V: Learning Principles from Multi-modal Human Preference

2025-05-24 · Jiayi Zhou, Jiaming Ji, Boyuan Chen, Jiapeng Sun 외

Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor int…

Blocking Out-of-Distribution Generalization

Prototypical Reward Network for Data-Efficient RLHF

2024-06-06 · Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen 외

The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to s…