Curiosity-Driven Reinforcement Learning from Human Feedback

2025-01-20 · Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output diversity. This trade-off between diversity and alignment quality remains a significant challenge. Drawing inspiration from curiosity-driven exploration in reinforcement learning, we introduce curiosity-driven RLHF (CD-RLHF), a framework that incorporates intrinsic rewards for novel states, alongside traditional sparse extrinsic rewards, to optimize both output diversity and alignment quality. We demonstrate the effectiveness of CD-RLHF through extensive experiments on a range of tasks, including text summarization and instruction following. Our approach achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF. We make our code publicly available at https://github.com/ernie-research/CD-RLHF.

📄 PDF Abstract BibTeX arXiv:2501.11463

Code (1)

ernie-research/cd-rlhf 공식 구현 pytorch

Tasks

DiversityInstruction Following reinforcement-learning Reinforcement Learning Text Summarization

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

2019-10-23 · Patrik Reizinger, Márton Szemenyei

Reinforcement Learning enables to train an agent via interaction with the environment. However, in the majority of real-world scenarios, the extrinsic feedback is sparse or not sufficient, thus intrinsic reward formulati…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks

2022-01-20 · Chenyu Sun, Hangwei Qian, Chunyan Miao

Psychological curiosity plays a significant role in human intelligence to enhance learning through exploration and information acquisition. In the Artificial Intelligence (AI) community, artificial curiosity provides a n…

Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation

2019-08-01 · Yadan Luo, Zi Huang, Zheng Zhang, Ziwei Wang 외

Visual paragraph generation aims to automatically describe a given image from different perspectives and organize sentences in a coherent way. In this paper, we address three critical challenges for this task in a reinfo…

Decision Making Imitation Learning reinforcement-learning Reinforcement Learning+1

Curiosity-Driven Recommendation Strategy for Adaptive Learning via Deep Reinforcement Learning

2019-10-12 · Ruijian Han, Kani Chen, Chunxi Tan

The design of recommendations strategies in the adaptive learning system focuses on utilizing currently available information to provide individual-specific learning instructions for learners. As a critical motivate for …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Curiosity-Driven Multi-Criteria Hindsight Experience Replay

2019-06-09 · John B. Lanier, Stephen Mcaleer, Pierre Baldi

Dealing with sparse rewards is a longstanding challenge in reinforcement learning. The recent use of hindsight methods have achieved success on a variety of sparse-reward tasks, but they fail on complex tasks such as sta…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)