On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
We consider the problem of using expert data with unobserved confounders for imitation and reinforcement learning. We begin by defining the problem of learning from confounded expert data in a contextual MDP setup. We analyze the limitations of learning from such data with and without external reward, and propose an adjustment of standard imitation learning algorithms to fit this setup. We then discuss the problem of distribution shift between the expert data and the online environment when the data is only partially observable. We prove possibility and impossibility results for imitation learning under arbitrary distribution shift of the missing covariates. When additional external reward is provided, we propose a sampling procedure that addresses the unknown shift and prove convergence to an optimal solution. Finally, we validate our claims empirically on challenging assistive healthcare and recommender system simulation tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningRecommendation Systemsreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Proxy Methods for Domain Adaptation
We study the problem of domain adaptation under distribution shift, where the shift is due to a change in the distribution of an unobserved, latent variable that confounds both the covariates and the labels. In this sett…
Domain AdaptationPoint-Identification of a Robust Predictor Under Latent Shift with Imperfect Proxies
Addressing the domain adaptation problem becomes more challenging when distribution shifts across domains stem from latent confounders that affect both covariates and outcomes. Existing proxy-based approaches that addres…
Domain AdaptationActive LearningDITTO: Offline Imitation Learning with World Models
We propose DITTO, an offline imitation learning algorithm which uses world models and on-policy reinforcement learning to addresses the problem of covariate shift, without access to an oracle or any additional online int…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Causal Inference using Gaussian Processes with Structured Latent Confounders
Latent confounders---unobserved variables that influence both treatment and outcome---can bias estimates of causal effects. In some cases, these confounders are shared across observations, e.g. all students taking a cour…
Causal InferenceGaussian ProcessesTowards Backwards-Compatible Data with Confounded Domain Adaptation
Most current domain adaptation methods address either covariate shift or label shift, but are not applicable where they occur simultaneously and are confounded with each other. Domain adaptation approaches which do accou…
Domain Adaptation