Online Apprenticeship Learning

2021-02-13 · Lior Shani, Tom Zahavy, Shie Mannor

In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. Instead, we observe trajectories sampled by an expert that acts according to some policy. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We introduce an online variant of AL (Online Apprenticeship Learning; OAL), where the agent is expected to perform comparably to the expert while interacting with the environment. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms: one for policy optimization and another for learning the worst case cost. By employing optimistic exploration, we derive a convergent algorithm with $O(\sqrt{K})$ regret, where $K$ is the number of interactions with the MDP, and an additional linear error term that depends on the amount of expert trajectories available. Importantly, our algorithm avoids the need to solve an MDP at each iteration, making it more practical compared to prior AL methods. Finally, we implement a deep variant of our algorithm which shares some similarities to GAIL \cite{ho2016generative}, but where the discriminator is replaced with the costs learned by the OAL problem. Our simulations suggest that OAL performs well in high dimensional control problems.

📄 PDF Abstract BibTeX arXiv:2102.06924

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

GAIL Generative Adversarial Imitation Learning presents a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning…

Blending Autonomous Exploration and Apprenticeship Learning

2011-12-01 · NeurIPS 2011 12 · Thomas J. Walsh, Daniel K. Hewlett, Clayton T. Morrison

We present theoretical and empirical results for a framework that combines the benefits of apprenticeship and autonomous reinforcement learning. Our approach modifies an existing apprenticeship learning framework that …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs

2024-05-21 · Wengxi Li, Roy Pea, Nick Haber, Hari Subramonyam

Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challe…

Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations

2019-06-14 · NeurIPS 2020 12 · Rohan Paleja, Andrew Silva, Letian Chen, Matthew Gombolay

Resource scheduling and coordination is an NP-hard optimization requiring an efficient allocation of agents to a set of tasks with upper- and lower bound temporal and resource constraints. Due to the large-scale and dyna…

Decision Making Scheduling

Unknown mixing times in apprenticeship and reinforcement learning

2019-05-23 · Tom Zahavy, Alon Cohen, Haim Kaplan, Yishay Mansour

We derive and analyze learning algorithms for apprenticeship learning, policy evaluation, and policy gradient for average reward criteria. Existing algorithms explicitly require an upper bound on the mixing time. In cont…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sequence-based Multimodal Apprenticeship Learning For Robot Perception and Decision Making

2017-02-24 · Fei Han, Xue Yang, Yu Zhang, Hao Zhang

Apprenticeship learning has recently attracted a wide attention due to its capability of allowing robots to learn physical tasks directly from demonstrations provided by human experts. Most previous techniques assumed th…

Decision Making