Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient

2020-07-03 · Yufei Wang, Tianwei Ni

Exploration-exploitation dilemma has long been a crucial issue in reinforcement learning. In this paper, we propose a new approach to automatically balance between these two. Our method is built upon the Soft Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances the original task reward and the policy entropy, and hence controls the trade-off between exploitation and exploration. It is empirically shown that SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2), which uses constrained optimization for automatic adjustment, has some limitations. The core of our method, namely Meta-SAC, is to use metagradient along with a novel meta objective to automatically tune the entropy temperature in SAC. We show that Meta-SAC achieves promising performances on several of the Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most challenging tasks, humanoid-v2.

📄 PDF Abstract BibTeX arXiv:2007.01932

Code (1)

twni2016/Meta-SAC 공식 구현 pytorch

Tasks

BenchmarkingMuJoCo

Methods 이 논문이 사용한 방법론

Dilated Convolution 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Average Pooling 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

SAC 설명 없음

Target Entropy Annealing for Discrete Soft Actor-Critic

2021-12-06 · Yaosheng Xu, Dailin Hu, Litian Liang, Stephen Mcaleer 외

Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in continuous action space settings. It uses the maximum entropy framework for efficiency and stability, and applies a heuristic temperature Lagrange t…

Atari Games Scheduling

Temperature check: theory and practice for training models with softmax-cross-entropy losses

2020-10-14 · Atish Agarwala, Jeffrey Pennington, Yann Dauphin, Sam Schoenholz

The softmax function combined with a cross-entropy loss is a principled approach to modeling probability distributions that has become ubiquitous in deep learning. The softmax function is defined by a lone hyperparameter…

Sentiment Analysis

Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning

2021-11-28 · Dailin Hu, Pieter Abbeel, Roy Fox

Maximum Entropy Reinforcement Learning (MaxEnt RL) algorithms such as Soft Q-Learning (SQL) and Soft Actor-Critic trade off reward and policy entropy, which has the potential to improve training stability and robustness.…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

CAT-SAC: Soft Actor-Critic with Curiosity-Aware Entropy Temperature

2021-01-01 · Junfan Lin, Changxin Huang, Xiaodan Liang, Liang Lin

The trade-off between exploration and exploitation has long been a crucial issue in reinforcement learning~(RL). Most of the existing RL methods handle this problem by adding action noise to the policies, such as the Sof…

MuJoCo Reinforcement Learning (RL)

Meta Knowledge Distillation

2022-02-16 · Jihao Liu, Boxiao Liu, Hongsheng Li, Yu Liu

Recent studies pointed out that knowledge distillation (KD) suffers from two degradation problems, the teacher-student gap and the incompatibility with strong data augmentations, making it not applicable to training stat…

Data Augmentation Image Classification Knowledge Distillation