Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior

2023-07-27 · NeurIPS 2023 11

We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. Our framework invokes low-level controllers - either learned or implicit in position-command control - to stabilize imitation around expert demonstrations. We show that with (a) a suitable low-level stability guarantee and (b) a powerful enough generative model as our imitation learner, pure supervised behavior cloning can generate trajectories matching the per-time step distribution of essentially arbitrary expert trajectories in an optimal transport cost. Our analysis relies on a stochastic continuity property of the learned policy we call "total variation continuity" (TVC). We then show that TVC can be ensured with minimal degradation of accuracy by combining a popular data-augmentation regimen with a novel algorithmic trick: adding augmentation noise at execution time. We instantiate our guarantees for policies parameterized by diffusion models and prove that if the learner accurately estimates the score of the (noise-augmented) expert policy, then the distribution of imitator trajectories is close to the demonstrator distribution in a natural optimal transport distance. Our analysis constructs intricate couplings between noise-augmented trajectories, a technique that may be of independent interest. We conclude by empirically validating our algorithmic recommendations, and discussing implications for future research directions for better behavior cloning with generative modeling.

📄 PDF Abstract BibTeX arXiv:2307.14619

Code (0)

등록된 구현이 없습니다.

Tasks

Data Augmentation

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Error Bounds of Imitating Policies and Environments

2020-10-22 · NeurIPS 2020 12 · Tian Xu, Ziniu Li, Yang Yu

Imitation learning trains a policy by mimicking expert demonstrations. Various imitation methods were proposed and empirically evaluated, meanwhile, their theoretical understanding needs further studies. In this paper, w…

Imitation Learning Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guarantees

2026-02-18 · Itamar Hadad, Guy Katz, Shahaf Bassan arxiv

*Automated circuit discovery* is a central tool in mechanistic interpretability for identifying the internal components of neural networks responsible for specific behaviors. While prior methods have made significant pro…

Robust Imitation Learning from Corrupted Demonstrations

2022-01-29 · Liu Liu, Ziyang Tang, Lanqing Li, Dijun Luo

We consider offline Imitation Learning from corrupted demonstrations where a constant fraction of data can be noise or even arbitrary outliers. Classical approaches such as Behavior Cloning assumes that demonstrations ar…

continuous-control Continuous Control Imitation Learning

Disagreement-Regularized Imitation Learning

2020-05-01 · ICLR 2020 1 · Kiante Brantley, Wen Sun, Mikael Henaff

We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance …

continuous-control Continuous Control Imitation Learning

Inverse Reinforcement Learning from Non-Stationary Learning Agents

2024-10-18 · Kavinayan P. Sivakumar, Yi Shen, Zachary Bell, Scott Nivison 외

In this paper, we study an inverse reinforcement learning problem that involves learning the reward function of a learning agent using trajectory data collected while this agent is learning its optimal policy. To address…

reinforcement-learning Reinforcement Learning