Sample-based Distributional Policy Gradient

2020-01-08 · Rahul Singh, Keuntaek Lee, Yongxin Chen

Distributional reinforcement learning (DRL) is a recent reinforcement learning framework whose success has been supported by various empirical studies. It relies on the key idea of replacing the expected return with the return distribution, which captures the intrinsic randomness of the long term rewards. Most of the existing literature on DRL focuses on problems with discrete action space and value based methods. In this work, motivated by applications in robotics with continuous action space control settings, we propose sample-based distributional policy gradient (SDPG) algorithm. It models the return distribution using samples via a reparameterization technique widely used in generative modeling and inference. We compare SDPG with the state-of-art policy gradient method in DRL, distributed distributional deterministic policy gradients (D4PG), which has demonstrated state-of-art performance. We apply SDPG and D4PG to multiple OpenAI Gym environments and observe that our algorithm shows better sample efficiency as well as higher reward for most tasks.

📄 PDF Abstract BibTeX arXiv:2001.02652

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement LearningOpenAI Gym reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

N-step Returns $n$-step Returns are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as: $$ R\_{t}^{(n)} =…

Prioritized Experience Replay Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay…

Adam 설명 없음

Batch Normalization 설명 없음

D4PG D4PG, or Distributed Distributional DDPG, is a policy gradient algorithm that extends upon the DDPG. The improvements include a…

PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm

2023-06-11 · Wensong Bai, Chao Zhang, Yichao Fu, Peilin Zhao 외

In this paper, we propose the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager. Spec…

Continuous Control Distributional Reinforcement Learning Efficient Exploration reinforcement-learning+1

PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods

2024-07-18 · WooJae Jeon, KangJun Lee, Jeewoo Lee

This paper introduces PG-Rainbow, a novel algorithm that incorporates a distributional reinforcement learning framework with a policy gradient algorithm. Existing policy gradient methods are sample inefficient and rely o…

Atari Games Decision Making Distributional Reinforcement Learning Policy Gradient Methods+2

Towards Provable Log Density Policy Gradient

2024-03-03 · Pulkit Katdare, Anant Joshi, Katherine Driggs-Campbell

Policy gradient methods are a vital ingredient behind the success of modern reinforcement learning. Modern policy gradient methods, although successful, introduce a residual error in gradient estimation. In this work, we…

Policy Gradient Methods reinforcement-learning Reinforcement Learning

Policy Gradient for Continuous-Time Robust Markov Decision Processes

2026-06-03 · Tanya Veeravalli, David M. Bossens, Atsushi Nitanda arxiv

The framework of robust Markov decision processes (RMDPs) allows the design of reinforcement learning agents that satisfy performance guarantees under worst-case transition dynamics. Traditional RMDPs consider discrete-t…

Reinforcement Learning

Implicit Distributional Reinforcement Learning

2020-07-13 · NeurIPS 2020 12 · Yuguang Yue, Zhendong Wang, Mingyuan Zhou

To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor-critic (IDAC) that consists of a distributional critic, built on two deep generator ne…

Distributional Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning+1