$q$-Munchausen Reinforcement Learning

2022-05-16 · Lingwei Zhu, Zheng Chen, Eiji Uchibe, Takamitsu Matsubara

The recently successful Munchausen Reinforcement Learning (M-RL) features implicit Kullback-Leibler (KL) regularization by augmenting the reward function with logarithm of the current stochastic policy. Though significant improvement has been shown with the Boltzmann softmax policy, when the Tsallis sparsemax policy is considered, the augmentation leads to a flat learning curve for almost every problem considered. We show that it is due to the mismatch between the conventional logarithm and the non-logarithmic (generalized) nature of Tsallis entropy. Drawing inspiration from the Tsallis statistics literature, we propose to correct the mismatch of M-RL with the help of $q$-logarithm/exponential functions. The proposed formulation leads to implicit Tsallis KL regularization under the maximum Tsallis entropy framework. We show such formulation of M-RL again achieves superior performance on benchmark problems and sheds light on more general M-RL with various entropic indices $q$.

📄 PDF Abstract BibTeX arXiv:2205.07467

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Sparsemax Sparsemax is a type of activation/output function similar to the traditional softmax, but able to output sparse probabilities.…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence

2023-09-21 · NeurIPS 2023 11

Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a…

Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence

2023-01-27 · Lingwei Zhu, Zheng Chen, Matthew Schlegel, Martha White

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning

2024-03-06 · Zida Wu, Mathieu Lauriere, Samuel Jia Cong Chua, Matthieu Geist 외

Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems, but learning Nash equilibria in MFGs remains a challenging task. In this paper, we propose a deep reinforcement learning (DRL) algorithm…

Deep Reinforcement Learning

Munchausen Reinforcement Learning

2020-07-28 · NeurIPS 2020 12 · Nino Vieillard, Olivier Pietquin, Matthieu Geist

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimat…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning

2025-09-03 · Zida Wu, Mathieu Lauriere, Matthieu Geist, Olivier Pietquin 외 arxiv

Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems. Yet, learning Nash equilibria in MFGs remains a challenging problem, particularly when the initial distribution is unknown …

Reinforcement Learning