An Alternative Softmax Operator for Reinforcement Learning

2016-12-16 · ICML 2017 8 · Kavosh Asadi, Michael L. Littman

A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a single maximum utility decision. The Boltzmann softmax operator is the most commonly used softmax operator in this setting, but we show that this operator is prone to misbehavior. In this work, we study a differentiable softmax operator that, among other properties, is a non-expansion ensuring a convergent behavior in learning and planning. We introduce a variant of SARSA algorithm that, by utilizing the new operator, computes a Boltzmann policy with a state-dependent temperature parameter. We show that the algorithm is convergent and that it performs favorably in practice.

📄 PDF Abstract BibTeX arXiv:1612.05628

Code (3)

grayswanai/nanogcg pytorch

sudh1976/ECE510_DeepLearning_Project pytorch

thompsonj/nanoGCG pytorch

Tasks

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning

2021-09-29 · Erfan Miahi, Revan MacQueen, Alex Ayoub, Abbas Masoumzadeh 외

Soft-greedy operators, namely $\varepsilon$-greedy and softmax, remain a common choice to induce a basic level of exploration for action-value methods in reinforcement learning. These operators, however, have a few criti…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning with Dynamic Boltzmann Softmax Updates

2019-03-14 · Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen 외

Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the n…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1

A Convergent Variant of the Boltzmann Softmax Operator in Reinforcement Learning

2018-09-27 · Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen 외

The Boltzmann softmax operator can trade-off well between exploration and exploitation according to current estimation in an exponential weighting scheme, which is a promising way to address the exploration-exploitation …

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1

Softmax is $1/2$-Lipschitz: A tight bound across all $\ell_p$ norms

2025-10-27 · Pravin Nair arxiv

The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustn…

Reinforcement Learning

Softmax Deep Double Deterministic Policy Gradients

2020-10-19 · NeurIPS 2020 12 · Ling Pan, Qingpeng Cai, Longbo Huang

A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Althou…

continuous-control Continuous Control