Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning

2021-11-28 · Dailin Hu, Pieter Abbeel, Roy Fox

Maximum Entropy Reinforcement Learning (MaxEnt RL) algorithms such as Soft Q-Learning (SQL) and Soft Actor-Critic trade off reward and policy entropy, which has the potential to improve training stability and robustness. Most MaxEnt RL methods, however, use a constant tradeoff coefficient (temperature), contrary to the intuition that the temperature should be high early in training to avoid overfitting to noisy value estimates and decrease later in training as we increasingly trust high value estimates to truly lead to good rewards. Moreover, our confidence in value estimates is state-dependent, increasing every time we use more evidence to update an estimate. In this paper, we present a simple state-based temperature scheduling approach, and instantiate it for SQL as Count-Based Soft Q-Learning (CBSQL). We evaluate our approach on a toy domain as well as in several Atari 2600 domains and show promising results.

📄 PDF Abstract BibTeX arXiv:2111.14204

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Scheduling

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning

2026-01-27 · Tongxi Wang, Zhuoyang Xia, Xinran Chen, Shan Liu arxiv

Real-world reinforcement learning often faces environment drift, but most existing methods rely on static entropy coefficients/target entropy, causing over-exploration during stable periods and under-exploration after dr…

Reinforcement Learning

Target Entropy Annealing for Discrete Soft Actor-Critic

2021-12-06 · Yaosheng Xu, Dailin Hu, Litian Liang, Stephen Mcaleer 외

Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in continuous action space settings. It uses the maximum entropy framework for efficiency and stability, and applies a heuristic temperature Lagrange t…

Atari Games Scheduling

Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL

2025-10-25 · Guojian Zhan, Likun Wang, Pengcheng Wang, Feihong Zhang 외 arxiv

Maximum entropy has become a mainstream off-policy reinforcement learning (RL) framework for balancing exploitation and exploration. However, two bottlenecks still limit further performance improvement: (1) non-stationar…

Reinforcement Learning OpenAI Gym

Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR

2025-10-10 · Haomin Zhuang, Yujun Zhou, Taicheng Guo, Yue Huang 외 arxiv

Reinforcement Learning has demonstrated substantial improvements in the reasoning abilities of Large Language Models (LLMs), exhibiting significant applicability across various domains. Recent research has identified tha…

Reinforcement Learning

Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control

2019-11-21 · Marta Sarrico, Kai Arulkumaran, Andrea Agostinelli, Pierre Richemond 외

Deep networks have enabled reinforcement learning to scale to more complex and challenging domains, but these methods typically require large quantities of training data. An alternative is to use sample-efficient episodi…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)