Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning

2022-05-16 · Lingwei Zhu, Zheng Chen, Eiji Uchibe, Takamitsu Matsubara

Maximum Tsallis entropy (MTE) framework in reinforcement learning has gained popularity recently by virtue of its flexible modeling choices including the widely used Shannon entropy and sparse entropy. However, non-Shannon entropies suffer from approximation error and subsequent underperformance either due to its sensitivity or the lack of closed-form policy expression. To improve the tradeoff between flexibility and empirical performance, we propose to strengthen their error-robustness by enforcing implicit Kullback-Leibler (KL) regularization in MTE motivated by Munchausen DQN (MDQN). We do so by drawing connection between MDQN and advantage learning, by which MDQN is shown to fail on generalizing to the MTE framework. The proposed method Tsallis Advantage Learning (TAL) is verified on extensive experiments to not only significantly improve upon Tsallis-DQN for various non-closed-form Tsallis entropies, but also exhibits comparable performance to state-of-the-art maximum Shannon entropy algorithms.

📄 PDF Abstract BibTeX arXiv:2205.07885

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

$q$-Munchausen Reinforcement Learning

2022-05-16 · Lingwei Zhu, Zheng Chen, Eiji Uchibe, Takamitsu Matsubara

The recently successful Munchausen Reinforcement Learning (M-RL) features implicit Kullback-Leibler (KL) regularization by augmenting the reward function with logarithm of the current stochastic policy. Though significan…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning

2019-01-31 · Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi 외

In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP pro…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Tsallis Entropy Regularization for Linearly Solvable MDP and Linear Quadratic Regulator

2024-03-04 · Yota Hashizume, Koshi Oishi, Kenji Kashima

Shannon entropy regularization is widely adopted in optimal control due to its ability to promote exploration and enhance robustness, e.g., maximum entropy reinforcement learning known as Soft Actor-Critic. In this paper…

reinforcement-learning Reinforcement Learning

Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy

2024-07-04 · Lijun Bo, YiJie Huang, Xiang Yu, Tingting Zhang

This paper studies the continuous-time reinforcement learning in jump-diffusion models by featuring the q-learning (the continuous-time counterpart of Q-learning) under Tsallis entropy regularization. Contrary to the Sha…

Q-Learning

Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization

2018-09-02 · Gang Chen, Yiming Peng, Mengjie Zhang

Recently deep reinforcement learning (DRL) has achieved outstanding success on solving many difficult and large-scale RL problems. However the high sample cost required for effective learning often makes DRL unaffordable…

Deep Reinforcement Learning Diversity Reinforcement Learning Reinforcement Learning (RL)