Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning

2021-09-29 · Erfan Miahi, Revan MacQueen, Alex Ayoub, Abbas Masoumzadeh, Martha White

Soft-greedy operators, namely $\varepsilon$-greedy and softmax, remain a common choice to induce a basic level of exploration for action-value methods in reinforcement learning. These operators, however, have a few critical limitations. In this work, we investigate a simple soft-greedy operator, which we call resmax, that takes actions proportionally to their suboptimality gap: the residual to the estimated maximal value. It is simple to use and ensures coverage of the state-space like $\varepsilon$-greedy, but focuses exploration more on potentially promising actions like softmax. Further, it does not concentrate probability as quickly as softmax, and so better avoids overemphasizing sub-optimal actions that appear high-valued during learning. Additionally, we prove it is a non-expansion for any fixed exploration hyperparameter, unlike the softmax policy which requires a state-action specific temperature to obtain a non-expansion (called mellowmax). We empirically validate that resmax is comparable to or outperforms $\varepsilon$-greedy and softmax across a variety of environments in tabular and deep RL.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multiple-Step Greedy Policies in Approximate and Online Reinforcement Learning

2018-12-01 · NeurIPS 2018 12 · Yonathan Efroni, Gal Dalal, Bruno Scherrer, Shie Mannor

Multiple-step lookahead policies have demonstrated high empirical competence in Reinforcement Learning, via the use of Monte Carlo Tree Search or Model Predictive Control. In a recent work (Efroni et al., 2018), multiple…

Model Predictive Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning

2018-05-21 · Yonathan Efroni, Gal Dalal, Bruno Scherrer, Shie Mannor

Model Predictive Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

An Alternative Softmax Operator for Reinforcement Learning

2016-12-16 · ICML 2017 8 · Kavosh Asadi, Michael L. Littman

A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize …

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking

2025-08-05 · Junhong Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu 외 arxiv

Human cognition naturally engages with abstract and fluid concepts, whereas existing reasoning models often rely on generating discrete tokens, potentially constraining their expressive capabilities. Recent advancements …

Reinforcement Learning

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

2026-02-26 · Yuanjun Li, Bin Zhang, Hao Chen, Zhouyang Jiang 외 arxiv

Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation lead…

Multi-agent Reinforcement Learning