Policy Shaping: Integrating Human Feedback with Reinforcement Learning
A long term goal of Interactive Reinforcement Learning is to incorporate non-expert human feedback to solve complex tasks. State-of-the-art methods have approached this problem by mapping human information to reward and value signals to indicate preferences and then iterating over them to compute the necessary control policy. In this paper we argue for an alternate, more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct labels on the policy. We compare Advise to state-of-the-art approaches and highlight scenarios where it outperforms them and importantly is robust to infrequent and inconsistent human feedback.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Influencing Reinforcement Learning through Natural Language Guidance
Interactive reinforcement learning agents use human feedback or instruction to help them learn in complex environments. Often, this feedback comes in the form of a discrete signal that is either positive or negative. Whi…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Learning Efficient Dialogue Policy from Demonstrations through Shaping
Training a task-oriented dialogue agent with reinforcement learning is prohibitively expensive since it requires a large volume of interactions with users. Human demonstrations can be used to accelerate learning progress…
Domain AdaptationInteractive Reinforcement Learning for Table Balancing Robot
With the development of robotics, the use of robots in daily life is increasing, which has led to the need for anyone to easily train robots to improve robot use. Interactive reinforcement learning(IARL) is a method for …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Deep Reinforcement Learningreinforcement-learning+5Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
The correct specification of reward models is a well-known challenge in reinforcement learning. Hand-crafted reward functions often lead to inefficient or suboptimal policies and may not be aligned with user values. Rein…
HallucinationLanguage ModelingLanguage ModellingLarge Language Model+2Environment Shaping in Reinforcement Learning using State Abstraction
One of the central challenges faced by a reinforcement learning (RL) agent is to effectively learn a (near-)optimal policy in environments with large state spaces having sparse and noisy feedback signals. In real-world a…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)