GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning

2021-06-11 · Jiajun Fan, Changnan Xiao, Yue Huang

Deep Q Network (DQN) firstly kicked the door of deep reinforcement learning (DRL) via combining deep learning (DL) with reinforcement learning (RL), which has noticed that the distribution of the acquired data would change during the training process. DQN found this property might cause instability for training, so it proposed effective methods to handle the downside of the property. Instead of focusing on the unfavourable aspects, we find it critical for RL to ease the gap between the estimated data distribution and the ground truth data distribution while supervised learning (SL) fails to do so. From this new perspective, we extend the basic paradigm of RL called the Generalized Policy Iteration (GPI) into a more generalized version, which is called the Generalized Data Distribution Iteration (GDI). We see massive RL algorithms and techniques can be unified into the GDI paradigm, which can be considered as one of the special cases of GDI. We provide theoretical proof of why GDI is better than GPI and how it works. Several practical algorithms based on GDI have been proposed to verify the effectiveness and extensiveness of it. Empirical experiments prove our state-of-the-art (SOTA) performance on Arcade Learning Environment (ALE), wherein our algorithm has achieved 9620.98% mean human normalized score (HNS), 1146.39% median HNS and 22 human world record breakthroughs (HWRB) using only 200M training frames. Our work aims to lead the RL research to step into the journey of conquering the human world records and seek real superhuman agents on both performance and efficiency.

📄 PDF Abstract BibTeX arXiv:2106.06232

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

GDI: Rethinking What Makes Reinforcement Learning Different from Supervised Learning

2021-11-24 · AAAI Workshop ML4OR-22 2022 2 · Anonymous

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Rethinking Assumptions in Deep Anomaly Detection

2020-05-30 · Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Klaus-Robert Müller 외

Though anomaly detection (AD) can be viewed as a classification problem (nominal vs. anomalous) it is usually treated in an unsupervised manner since one typically does not have access to, or it is infeasible to utilize,…

Anomaly Detection

Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems

2020-09-21 · Findings of the Association for Computational Linguistics 2020 · Ziming Li, Julia Kiseleva, Maarten de Rijke

Dialogue policy learning for task-oriented dialogue systems has enjoyed great progress recently mostly through employing reinforcement learning methods. However, these approaches have become very sophisticated. It is tim…

Decoder Multi-Label Classification MUlTI-LABEL-ClASSIFICATION reinforcement-learning+3

General Intelligence Requires Rethinking Exploration

2022-11-15 · Minqi Jiang, Tim Rocktäschel, Edward Grefenstette

We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely sol…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

2026-04-08 · Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao 외 arxiv

A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervi…

Reinforcement Learning Domain Generalization