Modified Actor-Critics

2019-07-02 · Erinc Merdivan, Sten Hanke, Matthieu Geist

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are naturally on-policy. In this paper, we propose to combine (any kind of) soft greediness with Modified Policy Iteration (MPI). The proposed abstract framework applies repeatedly: (i) a partial policy evaluation step that allows off-policy learning and (ii) any softened greedy step. Our contribution can be seen as a new generic tool for the deep reinforcement learning toolbox. As a proof of concept, we instantiate this framework with the PPO greediness. Comparison to the original PPO shows that our algorithm is much more sample efficient. We also show that it is competitive with the state-of-art off-policy algorithm Soft Actor Critic (SAC).

📄 PDF Abstract BibTeX arXiv:1907.01298

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Entropy Regularization 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Adam 설명 없음

Soft Actor Critic Soft Actor Critic, or SAC, is an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims…

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

OSMAN â€• A Novel Arabic Readability Metric

2016-05-01 · LREC 2016 5 · Mahmoud El-Haj, Paul Rayson

We present OSMAN (Open Source Metric for Measuring Arabic Narratives) - a novel open source Arabic readability metric and tool. It allows researchers to calculate readability for Arabic text with and without diacritics. …

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

2019-03-11 · Denis Steckelmacher, Hélène Plisnier, Diederik M. Roijers, Ann Nowé

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling

Projected Natural Actor-Critic

2013-12-01 · NeurIPS 2013 12 · Philip S. Thomas, William C. Dabney, Stephen Giguere, Sridhar Mahadevan

Natural actor-critics are a popular class of policy search algorithms for finding locally optimal policies for Markov decision processes. In this paper we address a drawback of natural actor-critics that limits their rea…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient Continuous Control with Double Actors and Regularized Critics

2021-06-06 · Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Xiu Li

How to obtain good value estimation is one of the key problems in Reinforcement Learning (RL). Current value estimation methods, such as DDPG and TD3, suffer from unnecessary over- or underestimation bias. In this paper,…

continuous-control Continuous Control Reinforcement Learning (RL)

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

2026-02-19 · Nathan Samuel de Lara, Florian Shkurti arxiv

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide…

Reinforcement Learning Offline RL