Reward learning from human preferences and demonstrations in Atari
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.
Code (2)
Tasks
Atari GamesDeep Reinforcement LearningImitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Batch Reinforcement Learning from Crowds
A shortcoming of batch reinforcement learning is its requirement for rewards in data, thus not applicable to tasks without reward functions. Existing settings for lack of reward, such as behavioral cloning, rely on optim…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performanc…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Towards Consistent Performance on Atari using Expert Demonstrations
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify th…
Atari GamesDeep Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
Bayesian reward learning from demonstrations enables rigorous safety and uncertainty analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally intractable for co…
Atari GamesBayesian InferenceImitation LearningQ-based Variational Inverse Reinforcement Learning
The development of safe and beneficial AI requires that systems can learn and act in accordance with human preferences. However, explicitly specifying these preferences by hand is often infeasible. Inverse reinforcement …
Reinforcement LearningActive LearningAtari Games