Revisiting Discrete Soft Actor-Critic

2022-09-21 · Haibin Zhou, Tong Wei, Zichuan Lin, Junyou Li, Junliang Xing, Yuanchun Shi, Li Shen, Chao Yu, Deheng Ye

We study the adaption of Soft Actor-Critic (SAC), which is considered as a state-of-the-art reinforcement learning (RL) algorithm, from continuous action space to discrete action space. We revisit vanilla discrete SAC and provide an in-depth understanding of its Q value underestimation and performance instability issues when applied to discrete settings. We thereby propose Stable Discrete SAC (SDSAC), an algorithm that leverages entropy-penalty and double average Q-learning with Q-clip to address these issues. Extensive experiments on typical benchmarks with discrete action space, including Atari games and a large-scale MOBA game, show the efficacy of our proposed method. Our code is at: https://github.com/coldsummerday/SD-SAC.git.

📄 PDF Abstract BibTeX arXiv:2209.10081

Code (1)

coldsummerday/SD-SAC 공식 구현 pytorch

Tasks

Atari GamesQ-LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dilated Convolution 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Average Pooling 설명 없음

SAC 설명 없음

Soft Actor-Critic for Discrete Action Settings

2019-10-16 · Petros Christodoulou

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and s…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Soft Decomposed Policy-Critic: Bridging the Gap for Effective Continuous Control with Discrete RL

2023-08-20 · Yechen Zhang, Jian Sun, Gang Wang, Zhuo Li 외

Discrete reinforcement learning (RL) algorithms have demonstrated exceptional performance in solving sequential decision tasks with discrete action spaces, such as Atari games. However, their effectiveness is hindered wh…

Atari Games continuous-control Continuous Control Reinforcement Learning (RL)

DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic

2023-10-26 · Dexter Neo, Tsuhan Chen

We present a novel extension to the family of Soft Actor-Critic (SAC) algorithms. We argue that based on the Maximum Entropy Principle, discrete SAC can be further improved via additional statistical constraints derived …

Atari Games reinforcement-learning

Variational Quantum Soft Actor-Critic

2021-12-20 · Qingfeng Lan

Quantum computing has a superior advantage in tackling specific problems, such as integer factorization and Simon's problem. For more general tasks in machine learning, by applying variational quantum circuits, more and …

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Dissecting Discrete Soft Actor-Critic: Limitations and Principled Alternatives

2025-09-11 · Reza Asad, Reza Babanezhad, Sharan Vaswani arxiv

While Soft Actor-Critic (SAC) is highly effective in continuous control, its discrete counterpart (DSAC) performs poorly on challenging discrete-action domains such as Atari. Consequently, starting from DSAC, we revisit …

Continuous Control Atari Games