홈 › Papers

Reward Generalization in RLHF: A Topological Perspective

2024-02-15 · Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang

Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of reward generalization in reinforcement learning from human feedback (RLHF), focusing on the topology of information flow at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present induced Bayesian networks to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose reward modeling from tree-structured preference information. It is shown to reduce reward uncertainty by up to $\Theta(\log n/\log\log n)$ times compared to baselines, where $n$ is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65% against baselines, thus improving reward generalization for free via topology design, while reducing the amount of data requiring annotation.

📄 PDF Abstract BibTeX arXiv:2402.10184

Code (0)

등록된 구현이 없습니다.

Tasks

Generalization BoundsLanguage ModellingLarge Language Model

Reinforcement Learning from Human Feedback: A Statistical Perspective

2026-04-02 · Pangpang Liu, Chengchun Shi, Will Wei Sun arxiv

Reinforcement learning from human feedback (RLHF) has emerged as a central framework for aligning large language models (LLMs) with human preferences. Despite its practical success, RLHF raises fundamental statistical qu…

Reinforcement Learning Active Learning

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

2024-02-14 · Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao 외

Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models with human values, reward hacking, also termed reward overoptimization, remains a critical challenge. This issue primar…

Secrets of RLHF in Large Language Models Part II: Reward Modeling

2024-01-11 · Binghai Wang, Rui Zheng, Lu Chen, Yan Liu 외

Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward mode…

Contrastive Learning Meta-Learning reinforcement-learning

Generative RLHF-V: Learning Principles from Multi-modal Human Preference

2025-05-24 · Jiayi Zhou, Jiaming Ji, Boyuan Chen, Jiapeng Sun 외

Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor int…

Blocking Out-of-Distribution Generalization

Towards a Theoretical Understanding to the Generalization of RLHF

2026-01-23 · Zhaochun Li, Mingyang Yi, Yue Wang, Shisheng Cui 외 arxiv

Reinforcement Learning from Human Feedback (RLHF) and its variants have emerged as the dominant approaches for aligning Large Language Models with human intent. While empirically effective, the theoretical generalization…

Reinforcement Learning