RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability
Visual model-based RL methods typically encode image observations into low-dimensional representations in a manner that does not eliminate redundant information. This leaves them susceptible to spurious variations -- changes in task-irrelevant components such as background distractors or lighting conditions. In this paper, we propose a visual model-based RL method that learns a latent representation resilient to such spurious variations. Our training objective encourages the representation to be maximally predictive of dynamics and reward, while constraining the information flow from the observation to the latent representation. We demonstrate that this objective significantly bolsters the resilience of visual model-based RL methods to visual distractors, allowing them to operate in dynamic environments. We then show that while the learned encoder is resilient to spirious variations, it is not invariant under significant distribution shift. To address this, we propose a simple reward-free alignment procedure that enables test time adaptation of the encoder. This allows for quick adaptation to widely differing environments without having to relearn the dynamics and policy. Our effort is a step towards making model-based RL a practical and useful tool for dynamic, diverse domains. We show its effectiveness in simulation benchmarks with significant spurious variations as well as a real-world egocentric navigation task with noisy TVs in the background. Videos and code at https://zchuning.github.io/repo-website/.
Code (1)
Tasks
Model-based Reinforcement Learningreinforcement-learningTest-time AdaptationSimilar Papers 제목 키워드 기반
Generative Posterior Networks for Approximately Bayesian Epistemic Uncertainty Estimation
In many real-world problems, there is a limited set of training data, but an abundance of unlabeled data. We propose a new method, Generative Posterior Networks (GPNs), that uses unlabeled data to estimate epistemic unce…
Ineffectiveness of Dictionary Coding to Infer Predictability Limits of Human Mobility
Recently, a series of models have been proposed to predict future movements of people. Meanwhile, dictionary coding algorithms are used to estimate the predictability limit of human mobility. Although dictionary coding i…
Overfitting and Generalizing with (PAC) Bayesian Prediction in Noisy Binary Classification
We consider a PAC-Bayes type learning rule for binary classification, balancing the training error of a randomized ''posterior'' predictor with its KL divergence to a pre-specified ''prior''. This can be seen as an exten…
Binary ClassificationGaussian Mean Field Regularizes by Limiting Learned Information
Variational inference with a factorized Gaussian posterior estimate is a widely used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In…
Variational InferenceComputerized Tomography with Total Variation and with Shearlets
To reduce the x-ray dose in computerized tomography (CT), many constrained optimization approaches have been proposed aiming at minimizing a regularizing function that measures lack of consistency with some prior knowled…