AFU: Actor-Free critic Updates in off-policy RL for continuous control

2024-04-24 · Nicolas Perrin-Gilbert

This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then study a simple failure mode of SAC and show how AFU can be modified to make actor updates less likely to become trapped in local optima, resulting in a second version of the algorithm, AFU-beta. Experimental results demonstrate the sample efficiency of both versions of AFU, marking it as the first model-free off-policy algorithm competitive with state-of-the-art actor-critic methods while departing from the actor-critic perspective.

📄 PDF Abstract BibTeX arXiv:2404.16159

Code (1)

perrin-isir/afu 공식 구현 jax

Tasks

continuous-control Continuous ControlQ-Learning

Methods 이 논문이 사용한 방법론

Dilated Convolution 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Average Pooling 설명 없음

SAC 설명 없음

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Guide Actor-Critic for Continuous Control

2017-05-22 · ICLR 2018 1 · Voot Tangkaratt, Abbas Abdolmaleki, Masashi Sugiyama

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Risk-Sensitive Exponential Actor Critic

2026-02-06 · Alonso Granados, Jason Pacheco arxiv

Model-free deep reinforcement learning (RL) algorithms have achieved tremendous success on a range of challenging tasks. However, safety concerns remain when these methods are deployed on real-world applications, necessi…

Reinforcement Learning

Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning

2025-01-29 · Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, Doina Precup

Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by t…

continuous-control Continuous Control Efficient Exploration Reinforcement Learning (RL)+1

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

2019-06-10 · Che Wang, Keith Ross

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement learning (DRL) algorithm based on maximum entropy reinforcement learning. By combining off-policy updates with an actor-critic formulation, SAC ach…

Deep Reinforcement Learning MuJoCo Q-Learning reinforcement-learning+2

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

2019-03-11 · Denis Steckelmacher, Hélène Plisnier, Diederik M. Roijers, Ann Nowé

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling