Invariance to Quantile Selection in Distributional Continuous Control
In recent years distributional reinforcement learning has produced many state of the art results. Increasingly sample efficient Distributional algorithms for the discrete action domain have been developed over time that vary primarily in the way they parameterize their approximations of value distributions, and how they quantify the differences between those distributions. In this work we transfer three of the most well-known and successful of those algorithms (QR-DQN, IQN and FQF) to the continuous action domain by extending two powerful actor-critic algorithms (TD3 and SAC) with distributional critics. We investigate whether the relative performance of the methods for the discrete action space translates to the continuous case. To that end we compare them empirically on the pybullet implementations of a set of continuous control tasks. Our results indicate qualitative invariance regarding the number and placement of distributional atoms in the deterministic, continuous action setting.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlDistributional Reinforcement LearningSimilar Papers 제목 키워드 기반
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Qu…
continuous-controlContinuous ControlDFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training
Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL …
Reinforcement LearningExploration with Multi-Sample Target Values for Distributional Reinforcement Learning
Distributional reinforcement learning (RL) aims to learn a value-network that predicts the full distribution of the returns for a given state, often modeled via a quantile-based critic. This approach has been successfull…
continuous-controlContinuous ControlDistributional Reinforcement LearningHumanoid Control+3Distributional Reinforcement Learning with Monotonic Splines
Distributional Reinforcement Learning (RL) differs from traditional RL by estimating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to param…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Fully Parameterized Quantile Function for Distributional Reinforcement Learning
Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The k…
Atari GamesDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+1