Reinforcement Learning with Dynamic Boltzmann Softmax Updates

2019-03-14 · Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, Longbo Huang, Tie-Yan Liu

Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this paper, we propose to update the value function with dynamic Boltzmann softmax (DBS) operator, which has good convergence property in the setting of planning and learning. Experimental results on GridWorld show that the DBS operator enables better estimation of the value function, which rectifies the convergence issue of the softmax operator. Finally, we propose the DBS-DQN algorithm by applying dynamic Boltzmann softmax updates in deep Q-network, which outperforms DQN substantially in 40 out of 49 Atari games.

📄 PDF Abstract BibTeX arXiv:1903.05926

Code (1)

Arrabonae/openai_DDDQN pytorch

Tasks

Atari GamesQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

A Convergent Variant of the Boltzmann Softmax Operator in Reinforcement Learning

2018-09-27 · Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen 외

The Boltzmann softmax operator can trade-off well between exploration and exploitation according to current estimation in an exponential weighting scheme, which is a promising way to address the exploration-exploitation …

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1

An Alternative Softmax Operator for Reinforcement Learning

2016-12-16 · ICML 2017 8 · Kavosh Asadi, Michael L. Littman

A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize …

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Computable Fairness: Boltzmann-Softmax Control for AI Resource Allocation

2026-04-12 · Ji-Won Park, Chae Un Kim arxiv

In large-scale AI systems, allocating scarce resources such as GPU compute time and bandwidth among multiple agents is a critical challenge. Conventional policies focus on efficiency metrics, potentially leading to domin…

Hyperproperty-Constrained Secure Reinforcement Learning

2025-07-31 · Ernest Bonnah, Luan Viet Nguyen, Khaza Anuarul Hoque arxiv

Hyperproperties for Time Window Temporal Logic (HyperTWTL) is a domain-specific formal specification language known for its effectiveness in compactly representing security, opacity, and concurrency properties for roboti…

Reinforcement Learning

Softmax Deep Double Deterministic Policy Gradients

2020-10-19 · NeurIPS 2020 12 · Ling Pan, Qingpeng Cai, Longbo Huang

A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Althou…

continuous-control Continuous Control