Generalization and Regularization in DQN

2018-09-29 · Jesse Farebrother, Marlos C. Machado, Michael Bowling

Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably similar environments. In this paper we propose a protocol to evaluate generalization in reinforcement learning through different modes of Atari 2600 games. With that protocol we assess the generalization capabilities of DQN, one of the most traditional deep reinforcement learning algorithms, and we provide evidence suggesting that DQN overspecializes to the training environment. We then comprehensively evaluate the impact of dropout and $\ell_2$ regularization, as well as the impact of reusing learned representations to improve the generalization capabilities of DQN. Despite regularization being largely underutilized in deep reinforcement learning, we show that it can, in fact, help DQN learn more general features. These features can be reused and fine-tuned on similar tasks, considerably improving DQN's sample efficiency.

📄 PDF Abstract BibTeX arXiv:1810.00123

Code (1)

jessefarebro/dqn-ale 공식 구현 tf

Tasks

Atari Games BenchmarkingDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Improving Generalization of Meta-Learning With Inverted Regularization at Inner-Level

2023-01-01 · CVPR 2023 1 · Lianzhe Wang, Shiji Zhou, Shanghang Zhang, Xu Chu 외

Despite the broad interest in meta-learning, the generalization problem remains one of the significant challenges in this field. Existing works focus on meta-generalization to unseen tasks at the meta-level by regula…

Meta-Learning

Revisiting Explicit Regularization in Neural Networks for Reliable Predictive Probability

2020-09-28 · Taejong Joo, Uijung Chung

From the statistical learning perspective, complexity control via explicit regularization is a necessity for improving the generalization of over-parameterized models, which deters the memorization of intricate patterns …

Memorization

MetaReg: Towards Domain Generalization using Meta-Regularization

2018-12-01 · NeurIPS 2018 12 · Yogesh Balaji, Swami Sankaranarayanan, Rama Chellappa

Training models that generalize to new domains at test time is a problem of fundamental importance in machine learning. In this work, we encode this notion of domain generalization using a novel regularization function. …

BIG-bench Machine Learning Domain Generalization Meta-Learning

Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

2019-05-30 · NeurIPS 2019 12 · Aditya Golatkar, Alessandro Achille, Stefano Soatto

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show th…

Data Augmentation

Adversarially robust generalization theory via Jacobian regularization for deep neural networks

2024-12-17 · Dongya Wu, Xin Li

Powerful deep neural networks are vulnerable to adversarial attacks. To obtain adversarially robust models, researchers have separately developed adversarial training and Jacobian regularization techniques. There are abu…

Adversarial Attack