Value Improved Actor Critic Algorithms

2024-06-03 · Yaniv Oren, Moritz A. Zanger, Pascal R. van der Vaart, Matthijs T. J. Spaan, Wendelin Bohmer

Many modern reinforcement learning algorithms build on the actor-critic (AC) framework: iterative improvement of a policy (the actor) using policy improvement operators and iterative approximation of the policy's value (the critic). In contrast, the popular value-based algorithm family employs improvement operators in the value update, to iteratively improve the value function directly. In this work, we propose a general extension to the AC framework that employs two separate improvement operators: one applied to the policy in the spirit of policy-based algorithms and one applied to the value in the spirit of value-based algorithms, which we dub Value-Improved AC (VI-AC). We design two practical VI-AC algorithms based in the popular online off-policy AC algorithms TD3 and DDPG. We evaluate VI-TD3 and VI-DDPG in the Mujoco benchmark and find that both improve upon or match the performance of their respective baselines in all environments tested.

📄 PDF Abstract BibTeX arXiv:2406.01423

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCo

Methods 이 논문이 사용한 방법론

Batch Normalization 설명 없음

Weight Decay 설명 없음

Target Policy Smoothing Target Policy Smoothing is a regularization strategy for the value function in reinforcement learning. Deterministic policies can overfit to narrow peaks in the value…

Adam 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Clipped Double Q-learning 설명 없음

TD3 TD3 builds on the DDPG algorithm for reinforcement learning, with a couple of modifications aimed at tackling overestimation bias…

Multi-State TD Target for Model-Free Reinforcement Learning

2024-05-26 · Wuhao Wang, Zhiyong Chen, Lepeng Zhang

Temporal difference (TD) learning is a fundamental technique in reinforcement learning that updates value estimates for states or state-action pairs using a TD target. This target represents an improved estimate of the t…

Management reinforcement-learning Reinforcement Learning

Decision-Aware Model Learning for Actor-Critic Methods: When Theory Does Not Meet Practice

2020-10-19 · NeurIPS Workshop ICBINB 2020 12 · Ângelo Gregório Lovatto, Thiago Pereira Bueno, Denis Mauá, Leliane Nunes de Barros

Actor-Critic methods are a prominent class of modern reinforcement learning algorithms based on the classic Policy Iteration procedure. Despite many successful cases, Actor-Critic methods tend to require a gigantic numbe…

Neural Machine Translation with Monte-Carlo Tree Search

2020-04-27 · Jerrod Parker, Jerry Zikun Chen

Recent algorithms in machine translation have included a value network to assist the policy network when deciding which word to output at each step of the translation. The addition of a value network helps the algorithm …

Machine Translation Translation

Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation

2018-02-21 · Hamid Reza Maei

We present the first class of policy-gradient algorithms that work with both state-value and policy function-approximation, and are guaranteed to converge under off-policy training. Our solution targets problems in reinf…

Reinforcement Learning

Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

2026-04-30 · Zhenjie Ren, Xiaoli Wei, Xiang Yu, Xun Yu Zhou arxiv

This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establis…