Learning Dynamics and Generalization in Reinforcement Learning

2022-06-05 · Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, Yarin Gal

Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this paper, we analyze the learning dynamics of temporal difference algorithms to gain novel insight into the tension between these two objectives. We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training, and at the same time induces the second-order effect of discouraging generalization. We corroborate these findings in deep RL agents trained on a range of environments, finding that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly initialized networks and networks trained with policy gradient methods. Finally, we investigate how post-training policy distillation may avoid this pitfall, and show that this approach improves generalization to novel environments in the ProcGen suite and improves robustness to input perturbations.

📄 PDF Abstract BibTeX arXiv:2206.02126

Code (0)

등록된 구현이 없습니다.

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

2020-05-14 · ICML 2020 1 · Kimin Lee, Younggyo Seo, Seung-Hyun Lee, Honglak Lee 외

Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment's dynamics. However, learning a global model that can generalize across differ…

model Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Generalization Through the Lens of Learning Dynamics

2022-12-11 · Clare Lyle

A machine learning (ML) system must learn not only to match the output of a target function on a training set, but also to generalize to novel situations in order to yield accurate predictions at deployment. In most prac…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

2020-10-26 · NeurIPS 2020 12 · Younggyo Seo, Kimin Lee, Ignasi Clavera, Thanard Kurutach 외

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in d…

Clustering Model-based Reinforcement Learning Multiple-choice Prediction+3

Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning

2023-10-09 · Fan-Ming Luo, Tian Xu, Xingchen Cao, Yang Yu

Learning a precise dynamics model can be crucial for offline reinforcement learning, which, unfortunately, has been found to be quite challenging. Dynamics models that are learned by fitting historical transitions often …

D4RL Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning

Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution

2023-02-19 · Jie Wang, Rui Yang, Zijie Geng, Zhihao Shi 외

Generalization in partially observed markov decision processes (POMDPs) is critical for successful applications of visual reinforcement learning (VRL) in real scenarios. A widely used idea is to learn task-relevant repre…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning