Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

2020-01-09 · Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Bo Cheng

In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q-value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q-value overestimations because it is capable of adaptively adjusting the update stepsize of the Q-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.

📄 PDF Abstract BibTeX arXiv:2001.02811

Code (3)

Jingliang-Duan/Distributional-Soft-Actor-Critic pytorch

MatthieuSarkis/Portfolio-Optimization-and-Goal-Based-Investment-with-Reinforcement-Learning pytorch

jingliang-duan/dsac-t pytorch

Tasks

continuous-control Continuous ControlMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic

2020-02-13 · Yangang Ren, Jingliang Duan, Shengbo Eben Li, Yang Guan 외

Reinforcement learning (RL) has achieved remarkable performance in numerous sequential decision making and control tasks. However, a common problem is that learned nearly optimal policy always overfits to the training en…

Autonomous Driving Autonomous Vehicles Decision Making reinforcement-learning+3

DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty

2025-06-14 · Mingxuan Cui, Duo Zhou, Yuxuan Han, Grani A. Hanasusanto 외

Deep reinforcement learning (RL) has achieved significant success, yet its application in real-world scenarios is often hindered by a lack of robustness to environmental uncertainties. To solve this challenge, some robus…

continuous-control Continuous Control Deep Reinforcement Learning Offline RL+3

Distributional Soft Actor-Critic with Diffusion Policy

2025-07-02 · Tong Liu, Yinuo Wang, Xujie Song, Wenjun Zou 외 arxiv

Reinforcement learning has been proven to be highly effective in handling complex control tasks. Traditional methods typically use unimodal distributions, such as Gaussian distributions, to model the output of value dist…

Reinforcement Learning

On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration

2025-01-22 · Yirui Zhou, Xiaowei Liu, Xiaofeng Zhang, Yangchun Zhang

This paper tackles the efficiency and stability issues in learning from observations (LfO). We commence by investigating how reward functions and policies generalize in LfO. Subsequently, the built-in reinforcement learn…

Reinforcement Learning (RL)

From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning

2025-10-07 · Li Zeqiao, Wang Yijing, Wang Haoyu, Li Zheng 외 arxiv

Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human…

Reinforcement Learning Autonomous Driving