Gamma-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction
We introduce the gamma-model, a predictive model of environment dynamics with an infinite, probabilistic horizon. Replacing standard single-step models with gamma-models leads to generalizations of the procedures that form the foundation of model-based control, including the model rollout and model-based value estimation. The gamma-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the gamma-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.
Code (1)
Tasks
Generative Adversarial NetworkSimilar Papers 제목 키워드 기반
Generative Temporal Difference Learning for Infinite-Horizon Prediction
We introduce the $\gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with $\gamma$-models leads to generalizations of the procedures ce…
Generative Adversarial NetworkPredictionStatistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces
Distributional reinforcement learning (DRL) has achieved empirical success in various domains. One core task in DRL is distributional policy evaluation, which involves estimating the return distribution $\eta^\pi$ for a …
Distributional Reinforcement LearningQ-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP
A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has …
Q-LearningReinforcement LearningBreaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model
This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator). We first consider $\gamma$-discounted infinite-horizon Markov decision processes (MDPs) …
modelModel-based Reinforcement LearningReinforcement Learning (RL)On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes
We consider infinite-horizon stationary $\gamma$-discounted Markov Decision Processes, for which it is known that there exists a stationary optimal policy. Using Value and Policy Iteration with some error $\epsilon…