Active teacher selection for reinforcement learning from human feedback

2023-10-23 · Rachel Freedman, Justin Svegliato, Kyle Wray, Stuart Russell

Reinforcement learning from human feedback (RLHF) enables machine learning systems to learn objectives from human feedback. A core limitation of these systems is their assumption that all feedback comes from a single human teacher, despite querying a range of distinct teachers. We propose the Hidden Utility Bandit (HUB) framework to model differences in teacher rationality, expertise, and costliness, formalizing the problem of learning from multiple teachers. We develop a variety of solution algorithms and apply them to two real-world domains: paper recommendation systems and COVID-19 vaccine testing. We find that the Active Teacher Selection (ATS) algorithm outperforms baseline algorithms by actively selecting when and which teacher to query. The HUB framework and ATS algorithm demonstrate the importance of leveraging differences between teachers to learn accurate reward models, facilitating future research on active teacher selection for robust reward modeling.

📄 PDF Abstract BibTeX arXiv:2310.15288

Code (0)

등록된 구현이 없습니다.

Tasks

Recommendation Systems reinforcement-learning Reinforcement Learning

Dual Active Learning for Reinforcement Learning from Human Feedback

2024-10-03 · Pangpang Liu, Chengchun Shi, Will Wei Sun

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objec…

Active Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-trainer Interactive Reinforcement Learning System

2022-10-14 · Zhaori Guo, Timothy J. Norman, Enrico H. Gerding

Interactive reinforcement learning can effectively facilitate the agent training via human feedback. However, such methods often require the human teacher to know what is the correct action that the agent should take. In…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Rewards from Linguistic Feedback

2020-09-30 · Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan 외

We explore unconstrained natural language feedback as a learning signal for artificial agents. Humans use rich and varied language to teach, yet most prior work on interactive learning from language assumes a particular …

Aspect-Based Sentiment Analysis Aspect-Based Sentiment Analysis (ABSA)reinforcement-learning Reinforcement Learning+2

The Expertise Problem: Learning from Specialized Feedback

2022-11-12 · Oliver Daniels-Koch, Rachel Freedman

Reinforcement learning from human feedback (RLHF) is a powerful technique for training agents to perform difficult-to-specify tasks. However, human feedback can be noisy, particularly when human teachers lack relevant kn…

Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores

2023-07-11 · Shukai Liu, Chenming Wu, Ying Li, Liangjun Zhang

Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of a large amount of interactive feedback. This paper presents a …

reinforcement-learning Reinforcement Learning