Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage
This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO): an algorithmic framework that utilizes the static dataset to solve the offline IL problem efficiently both in theory and in practice. In theory, even if the behavior policy is highly sub-optimal compared to the expert, we show that as long as the data from the behavior policy provides sufficient coverage on the expert state-action traces (and with no necessity for a global coverage over the entire state-action space), MILO can provably combat the covariate shift issue in IL. Complementing our theory results, we also demonstrate that a practical implementation of our approach mitigates covariate shift on benchmark MuJoCo continuous control tasks. We demonstrate that with behavior policies whose performances are less than half of that of the expert, MILO still successfully imitates with an extremely low number of expert state-action pairs while traditional offline IL methods such as behavior cloning (BC) fail completely. Source code is provided at https://github.com/jdchang1/milo.
Code (1)
Tasks
continuous-controlContinuous ControlImitation LearningMuJoCoSimilar Papers 제목 키워드 기반
Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline datas…
continuous-controlContinuous ControlImitation LearningMuJoCoDITTO: Offline Imitation Learning with World Models
We propose DITTO, an offline imitation learning algorithm which uses world models and on-policy reinforcement learning to addresses the problem of covariate shift, without access to an oracle or any additional online int…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning
A pervasive phenomenon in machine learning applications is distribution shift, where training and deployment conditions for a machine learning model differ. As distribution shift typically results in a degradation in per…
regressionreinforcement-learningOffline Imitation Learning with Model-based Reverse Augmentation
In offline Imitation Learning (IL), one of the main challenges is the \textit{covariate shift} between the expert observations and the actual distribution encountered by the agent, because it is difficult to determine wh…
Imitation LearningmodelMitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models
We propose the use of latent space generative world models to address the covariate shift problem in autonomous driving. A world model is a neural network capable of predicting an agent's next state given past states and…
Autonomous DrivingAutonomous VehiclesImitation Learning