Disagreement-Regularized Imitation Learning
We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost. Unlike adversarial imitation methods, it uses a fixed reward function which is easy to optimize. We prove a regret bound for the algorithm which is linear in the time horizon multiplied by a coefficient which we show to be low for certain problems in which behavioral cloning fails. We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or significantly outperforms behavioral cloning and generative adversarial imitation learning.
Code (2)
Tasks
continuous-controlContinuous ControlImitation LearningSimilar Papers 제목 키워드 기반
Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL)
Machine-learning paradigms such as imitation learning and reinforcement learning can generate highly performant agents in a variety of complex environments. However, commonly used methods require large quantities of data…
Imitation LearningMuJoCoreinforcement-learningReinforcement LearningDisagreement-Regularized Importance Sampling for Adversarial Label Corruption
Standard Importance Sampling (IS) collapses under label corruption because high-norm examples, prioritized for variance reduction, are often adversarial outliers. We formalize this misalignment using an $\varepsilon$-con…
FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models
Aligning Large Language Models (LLMs) with human values often involves balancing multiple, conflicting objectives such as helpfulness and harmlessness. Training these models is computationally intensive, and centralizing…
Federated LearningSemi-supervised novelty detection using ensembles with regularized disagreement
Deep neural networks often predict samples with high confidence even when they come from unseen classes and should instead be flagged for expert evaluation. Current novelty detection algorithms cannot reliably identify s…
Novelty DetectionOut-of-Distribution DetectionOut of Distribution (OOD) DetectionImitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations. In contrast to Learning from Demonstration (LfD) that involves both action and state supervision, LfO i…
Imitation Learning