Distributional Advantage Actor-Critic
In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optimal action is then chosen to be the action a with the largest value estimated by value function. However, recent developments have shown both theoretical and experimental evidence of superior performance when value function is replaced with value distribution in context of deep Q learning [1]. In this paper, we develop a new algorithm that combines advantage actor-critic with value distribution estimated by quantile regression. We evaluated this new algorithm, termed Distributional Advantage Actor-Critic (DA2C or QR-A2C) on a variety of tasks, and observed it to achieve at least as good as baseline algorithms, and outperforming baseline in some tasks with smaller variance and increased stability.
Code (0)
등록된 구현이 없습니다.
Tasks
Q-Learningquantile regressionSimilar Papers 제목 키워드 기반
Distributional Reinforcement Learning via the Cramér Distance
This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cramér-based Distributional Sof…
Reinforcement LearningContinuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms
Distributed Distributional DrQ is a model-free and off-policy RL algorithm for continuous control tasks based on the state and observation of the agent, which is an actor-critic method with the data-augmentation and the …
continuous-controlContinuous ControlData Augmentationreinforcement-learning+1On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration
This paper tackles the efficiency and stability issues in learning from observations (LfO). We commence by investigating how reward functions and policies generalize in LfO. Subsequently, the built-in reinforcement learn…
Reinforcement Learning (RL)A Distributional Perspective on Actor-Critic Framework
Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, ac…
Distributional Reinforcement LearningD2 Actor Critic: Diffusion Actor Meets Distributional Critic
We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of…
Reinforcement Learning