Variance Control for Distributional Reinforcement Learning

2023-07-30 · Qi Kuang, Zhoufan Zhu, Liwen Zhang, Fan Zhou

Although distributional reinforcement learning (DRL) has been widely examined in the past few years, very few studies investigate the validity of the obtained Q-function estimator in the distributional setting. To fully understand how the approximation errors of the Q-function affect the whole training process, we do some error analysis and theoretically show how to reduce both the bias and the variance of the error terms. With this new understanding, we construct a new estimator \emph{Quantiled Expansion Mean} (QEM) and introduce a new DRL algorithm (QEMRL) from the statistical perspective. We extensively evaluate our QEMRL algorithm on a variety of Atari and Mujoco benchmark tasks and demonstrate that QEMRL achieves significant improvement over baseline algorithms in terms of sample efficiency and convergence performance.

📄 PDF Abstract BibTeX arXiv:2307.16152

Code (1)

kuangqi927/qem 공식 구현 pytorch

Tasks

Distributional Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning

Risk Perspective Exploration in Distributional Reinforcement Learning

2022-06-28 · Jihwan Oh, Joonkee Kim, Se-Young Yun

Distributional reinforcement learning demonstrates state-of-the-art performance in continuous and discrete control settings with the features of variance and risk, which can be used to explore. However, the exploration m…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Invariance to Quantile Selection in Distributional Continuous Control

2022-12-29 · Felix Grün, Muhammad Saif-ur-Rehman, Tobias Glasmachers, Ioannis Iossifidis

In recent years distributional reinforcement learning has produced many state of the art results. Increasingly sample efficient Distributional algorithms for the discrete action domain have been developed over time that …

continuous-control Continuous Control Distributional Reinforcement Learning

Path-Coupled Bellman Flows for Distributional Reinforcement Learning

2026-05-07 · Boyang Xu, Qing Zou, Siqin Yang, Hao Yan arxiv

Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundar…

Reinforcement Learning Offline RL

DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning

2020-04-30 · Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang 외

In this paper, we present a new reinforcement learning (RL) algorithm called Distributional Soft Actor Critic (DSAC), which exploits the distributional information of accumulated rewards to achieve better performance. Se…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Normality-Guided Distributional Reinforcement Learning for Continuous Control

2022-08-28 · Ju-Seung Byun, Andrew Perrault

Learning a predictive model of the mean return, or value function, plays a critical role in many reinforcement learning algorithms. Distributional reinforcement learning (DRL) has been shown to improve performance by mod…

continuous-control Continuous Control Distributional Reinforcement Learning reinforcement-learning+2