Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

2019-03-11 · Denis Steckelmacher, Hélène Plisnier, Diederik M. Roijers, Ann Nowé

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.

📄 PDF Abstract BibTeX arXiv:1903.04193

Code (1)

vub-ai-lab/bdpi 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

OPAC: Opportunistic Actor-Critic

2020-12-11 · Srinjoy Roy, Saptam Bakshi, Tamal Maharaj

Actor-critic methods, a type of model-free reinforcement learning (RL), have achieved state-of-the-art performances in many real-world domains in continuous control. Despite their success, the wide-scale deployment of th…

continuous-control Continuous Control MuJoCo Reinforcement Learning (RL)

Model Based Meta Learning of Critics for Policy Gradients

2022-04-05 · Sarah Bechtle, Ludovic Righetti, Franziska Meier

Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in re…

Meta-Learning

Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration

2025-09-26 · Qinxun Bai, Yuxuan Han, Wei Xu, Zhengyuan Zhou arxiv

The actor-critic (AC) framework has achieved strong empirical success in off-policy reinforcement learning but suffers from the "moving target" problem, where the evaluated policy changes continually. Functional critics,…

Reinforcement Learning

Balancing Value Underestimation and Overestimation with Realistic Actor-Critic

2021-10-19 · Sicen Li, Qinyun Tang, Yiming Pang, Xinmeng Ma 외

Model-free deep reinforcement learning (RL) has been successfully applied to challenging continuous control domains. However, poor sample efficiency prevents these methods from being widely used in real-world domains. Th…

continuous-control Continuous Control Deep Reinforcement Learning Efficient Exploration+3

Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction

2021-10-22 · Jiachen Li, Shuo Cheng, Zhenyu Liao, Huayan Wang 외

Improving the sample efficiency of reinforcement learning algorithms requires effective exploration. Following the principle of $\textit{optimism in the face of uncertainty}$ (OFU), we train a separate exploration policy…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1