ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations

2019-10-26 · Daniel Seita, David Chan, Roshan Rao, Chen Tang, Mandi Zhao, John Canny

Learning from demonstrations is a popular tool for accelerating and reducing the exploration requirements of reinforcement learning. When providing expert demonstrations to human students, we know that the demonstrations must fall within a particular range of difficulties called the "Zone of Proximal Development (ZPD)". If they are too easy the student learns nothing, but if they are too difficult the student is unable to follow along. This raises the question: Given a set of potential demonstrators, which among them is best suited for teaching any particular learner? Prior work, such as the popular Deep Q-learning from Demonstrations (DQfD) algorithm has generally focused on single demonstrators. In this work we consider the problem of choosing among multiple demonstrators of varying skill levels. Our results align with intuition from human learners: it is not always the best policy to draw demonstrations from the best performing demonstrator (in terms of reward). We show that careful selection of teaching strategies can result in sample efficiency gains in the learner's environment across nine Atari games

📄 PDF Abstract BibTeX arXiv:1910.12154

Code (2)

DanielTakeshi/dqn pytorch

RevanMacQueen/LearningFromHumans

Tasks

Atari GamesDeep Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

2018-05-20 · Daniel S. Brown, Scott Niekum

Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Teaching Inverse Reinforcement Learners via Features and Demonstrations

2018-10-21 · NeurIPS 2018 12 · Luis Haug, Sebastian Tschiatschek, Adish Singla

Learning near-optimal behaviour from an expert's demonstrations typically relies on the assumption that the learner knows the features that the true reward function depends on. In this paper, we study the problem of lear…

Reinforcement Learning

Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints

2019-06-02 · NeurIPS 2019 12 · Sebastian Tschiatschek, Ahana Ghosh, Luis Haug, Rati Devidze 외

Inverse reinforcement learning (IRL) enables an agent to learn complex behavior by observing demonstrations from a (near-)optimal policy. The typical assumption is that the learner's goal is to match the teacher's demons…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback

2023-09-16 · Rustam Zayanov, Francisco S. Melo, Manuel Lopes

We study the problem of teaching via demonstrations in sequential decision-making tasks. In particular, we focus on the situation when the teacher has no access to the learner's model and policy, and the feedback from th…

Active Learning Decision Making Sequential Decision Making

The Sample Complexity of Teaching-by-Reinforcement on Q-Learning

2020-06-16 · Xuezhou Zhang, Shubham Kumar Bharti, Yuzhe ma, Adish Singla 외

We study the sample complexity of teaching, termed as "teaching dimension" (TDim) in the literature, for the teaching-by-reinforcement paradigm, where the teacher guides the student through rewards. This is distinct from…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)