A Distributional Perspective on Actor-Critic Framework
Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and biased approximation. In this paper, we present a novel distributional actor-critic frame-work, GMAC, to address such problems. Adopting a stochastic policy removes the first two problems, and the bias in approximation is alleviated by minimizing the Cramer distance between the value distribution and its Bellman target distribution. In addition, GMAC improves data efficiency by generating the Bellman target distribution through Sample-Replacement algorithm, denoted by SR(λ), which provides a distributional generalization of multi-step policy evaluation algorithms.We empirically show that our method captures the multimodality of value distributions and improves the performance of conventional actor-critic methods with low computational cost in both discrete and continuous action spaces, using ArcadeLearning Environment (ALE) and PyBullet environment.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GMAC: A Distributional Perspective on Actor-Critic Framework
In this paper, we devise a distributional framework on actor-critic as a solution to distributional instability, action type restriction, and conflation between samples and statistics. We propose a new method that minimi…
Atari GamesContinuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms
Distributed Distributional DrQ is a model-free and off-policy RL algorithm for continuous control tasks based on the state and observation of the agent, which is an actor-critic method with the data-augmentation and the …
continuous-controlContinuous ControlData Augmentationreinforcement-learning+1Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism
Growth in the penetration of renewable energy sources makes supply more uncertain and leads to an increase in the system imbalance. This trend, together with the single imbalance pricing, opens an opportunity for balance…
Distributional Reinforcement LearningQ-Learningreinforcement-learningImplicit Distributional Reinforcement Learning
To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor-critic (IDAC) that consists of a distributional critic, built on two deep generator ne…
Distributional Reinforcement LearningOpenAI Gymreinforcement-learningReinforcement Learning+1DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning
In this paper, we present a new reinforcement learning (RL) algorithm called Distributional Soft Actor Critic (DSAC), which exploits the distributional information of accumulated rewards to achieve better performance. Se…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1