Measuring and Characterizing Generalization in Deep Reinforcement Learning

2018-12-07 · Sam Witty, Jun Ki Lee, Emma Tosch, Akanksha Atrey, Michael Littman, David Jensen

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.

📄 PDF Abstract BibTeX arXiv:1812.02868

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark

2021-03-29 · Sharada Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov 외

The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology

2021-01-19 · NeurIPS 2021 12 · David Pérez-Fernández, Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marta Villegas

Characterizing the structural properties of neural networks is crucial yet poorly understood, and there are no well-established similarity measures between networks. In this work, we observe that neural networks can be r…

Topological Data Analysis

Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning

2022-06-03 · Bertrand Charpentier, Ransalu Senanayake, Mykel Kochenderfer, Stephan Günnemann

Characterizing aleatoric and epistemic uncertainty on the predicted rewards can help in building reliable reinforcement learning (RL) systems. Aleatoric uncertainty results from the irreducible environment stochasticity …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

2021-01-13 · ICLR 2021 1 · Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, Marc G. Bellemare

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learni…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Characterizing the Action-Generalization Gap in Deep Q-Learning

2022-05-11 · Zhiyuan Zhou, Cameron Allen, Kavosh Asadi, George Konidaris

We study the action generalization ability of deep Q-learning in discrete action spaces. Generalization is crucial for efficient reinforcement learning (RL) because it allows agents to use knowledge learned from past exp…

Q-Learning Reinforcement Learning (RL)