What Would pi* Do?: Imitation Learning via Off-Policy Reinforcement Learning
Learning to imitate expert actions given demonstrations containing image observations is a difficult problem in robotic control. The key challenge is generalizing behavior to out-of-distribution states that differ from those in the demonstrations. State-of-the-art imitation learning algorithms perform well in environments with low-dimensional observations, but typically involve adversarial optimization procedures, which can be difficult to use with high-dimensional image observations. We propose a remarkably simple alternative based on off-policy soft Q-learning, which we call soft Q imitation learning (SQIL, pronounced "skill"), that rewards the agent for matching demonstrated actions in demonstrated states. The key idea is initially filling the agent's experience replay buffer with demonstrations, where rewards are set to a positive constant, and setting rewards to zero in all additional experiences. We derive SQIL from first principles as a method for performing approximate inference under the MaxCausalEnt model of expert behavior. The approximate inference objective trades off between a pure behavioral cloning loss and a regularization term that incorporates information about state transitions via the soft Bellman error. Our experiments show that SQIL matches the state of the art in low-dimensional environments, and significantly outperforms prior work in playing video games from high-dimensional images.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningQ-Learningreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Bad-Policy Density: A Measure of Reinforcement Learning Hardness
Reinforcement learning is hard in general. Yet, in many specific environments, learning is easy. What makes learning easy in one environment, but difficult in another? We address this question by proposing a simple measu…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments
Imitation learning (IL) enables agents to mimic expert behaviors. Most previous IL techniques focus on precisely imitating one policy through mass demonstrations. However, in many applications, what humans require is the…
Imitation LearningLearning When and What to Ask: a Hierarchical Reinforcement Learning Framework
Reliable AI agents should be mindful of the limits of their knowledge and consult humans when sensing that they do not have sufficient knowledge to make sound decisions. We formulate a hierarchical reinforcement learning…
Hierarchical Reinforcement Learningreinforcement-learningReinforcement Learning (RL)AI Exposure Scores: what they measure, what they miss, and what comes next
A set of exposure scores calculated in 2023 has become a central empirical input to the future of work debate. Produced by Eloundou et al. (2023) and referred to here as the GPTs are GPTs scores, they define exposure as …
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied …
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)