RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

2024-02-27 · Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, Fei-Yue Wang

Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL methods excessively depend on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method utilizes a sample selection-based discriminator to dynamically filter out noise and ensure robust training. To counteract the cumulative error stemming from incorrect selection, we suggest a warm start for the reward model, which additionally bridges the performance gap during the transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the state-of-the-art PbRL method. Code is available at https://github.com/CJReinforce/RIME_ICML2024.

📄 PDF Abstract BibTeX arXiv:2402.17257

Code (1)

cjreinforce/rime_icml2024 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning

On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

2025-05-30 · Soichiro Nishimori, Yu-Jie Zhang, Thanawat Lodkaew, Masashi Sugiyama

Optimizing policies based on human preferences is key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offlin…

Unbiased Alignment for Large Language Models with Noisy Preferences

2026-07-03 · Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu 외 arxiv

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the signif…

Reinforcement Learning

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

2025-05-15 · Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi 외

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human values. However, noisy preferences in human feedback can lead to reward misgeneralization - a phenomenon where reward models lea…

Bob and Alice Go to a Bar: Reasoning About Future With Probabilistic Programs

2021-08-09 · David Tolpin, Tomer Dobkin

It is well known that reinforcement learning can be cast as inference in an appropriate probabilistic model. However, this commonly involves introducing a distribution over agent trajectories with probabilities proportio…

Bayesian Inference Probabilistic Programming reinforcement-learning Reinforcement Learning+1

Adaptive Margin RLHF via Preference over Preferences

2025-09-26 · Yaswanth Chittepu, Prasann Singhal, Greg Durrett, Scott Niekum arxiv

Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLH…