Off-Policy Deep Reinforcement Learning without Exploration

2018-12-07 · Scott Fujimoto, David Meger, Doina Precup

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

📄 PDF Abstract BibTeX arXiv:1812.02900

Code (10)

sfujim/BCQ 공식 구현 pytorch

AurelianTactics/bcq_tensorflow tf

HzcIrving/DLRL-PlayGround/tree/main/Offline%20RL/BCQ pytorch

haosulab/ManiSkill-Learn pytorch

massquantity/DBRL pytorch

maziarg/PrivAttack-BCQ pytorch

sparkmxy/my-offlinerl pytorch

takuseno/d3rlpy pytorch

theSparta/off_policy_mujoco pytorch

thxsxth/POMDP_RLSepsis pytorch

Tasks

continuous-control Continuous ControlDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Weight Decay 설명 없음

Adam 설명 없음

Batch Normalization 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

DDPG DDPG, or Deep Deterministic Policy Gradient, is an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

2026-05-29 · Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada, Tadashi Kozuno 외 arxiv

In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greed…

Reinforcement Learning

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning

2024-10-31 · Jaeyoon Kim, Junyu Xuan, Christy Liang, Farookh Hussain

Offline-to-online reinforcement learning (RL) leverages both pre-trained offline policies and online policies trained for downstream tasks, aiming to improve data efficiency and accelerate performance enhancement. An exi…

Reinforcement Learning (RL)

Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning

2021-01-23 · William F. Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmaleki 외

Despite the close connection between exploration and sample efficiency, most state of the art reinforcement learning algorithms include no considerations for exploration beyond maximizing the entropy of the policy. In th…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Diverse Exploration for Fast and Safe Policy Improvement

2018-02-22 · Andrew Cohen, Lei Yu, Robert Wright

We study an important yet under-addressed problem of quickly and safely improving policies in online reinforcement learning domains. As its solution, we propose a novel exploration strategy - diverse exploration (DE), wh…

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Neurosymbolic Reinforcement Learning with Formally Verified Exploration

2020-09-26 · NeurIPS 2020 12 · Greg Anderson, Abhinav Verma, Isil Dillig, Swarat Chaudhuri

We present Revel, a partially neural reinforcement learning (RL) framework for provably safe exploration in continuous state and action spaces. A key challenge for provably safe deep RL is that repeatedly verifying neura…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration