Path Consistency Learning in Tsallis Entropy Regularized MDPs

2018-02-10 · ICML 2018 7 · Ofir Nachum, Yin-Lam Chow, Mohammad Ghavamzadeh

We study the sparse entropy-regularized reinforcement learning (ERL) problem in which the entropy term is a special form of the Tsallis entropy. The optimal policy of this formulation is sparse, i.e.,~at each state, it has non-zero probability for only a small number of actions. This addresses the main drawback of the standard Shannon entropy-regularized RL (soft ERL) formulation, in which the optimal policy is softmax, and thus, may assign a non-negligible probability mass to non-optimal actions. This problem is aggravated as the number of actions is increased. In this paper, we follow the work of Nachum et al. (2017) in the soft ERL setting, and propose a class of novel path consistency learning (PCL) algorithms, called {\em sparse PCL}, for the sparse ERL problem that can work with both on-policy and off-policy data. We first derive a {\em sparse consistency} equation that specifies a relationship between the optimal value function and policy of the sparse ERL along any system trajectory. Crucially, a weak form of the converse is also true, and we quantify the sub-optimality of a policy which satisfies sparse consistency, and show that as we increase the number of actions, this sub-optimality is better than that of the soft ERL optimal policy. We then use this result to derive the sparse PCL algorithms. We empirically compare sparse PCL with its soft counterpart, and show its advantage, especially in problems with a large number of actions.

📄 PDF Abstract BibTeX arXiv:1802.03501

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning

2019-01-31 · Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi 외

In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP pro…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning

2017-09-19 · Kyungjae Lee, Sungjoon Choi, Songhwai Oh

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization

2021-10-17 · Donghao Ying, Yuhao Ding, Javad Lavaei

We study entropy-regularized constrained Markov decision processes (CMDPs) under the soft-max parameterization, in which an agent aims to maximize the entropy-regularized value function while satisfying constraints on th…

Entropy-Regularized Partially Observed Markov Decision Processes

2021-12-22 · Timothy L. Molloy, Girish N. Nair

We investigate partially observed Markov decision processes (POMDPs) with cost functions regularized by entropy terms describing state, observation, and control uncertainty. Standard POMDP techniques are shown to offer b…

State Estimation

Beyond Softmax and Entropy: Convergence Rates of Policy Gradients with f-SoftArgmax Parameterization & Coupled Regularization

2026-01-18 · Safwan Labbi, Daniil Tiapkin, Paul Mangold, Eric Moulines arxiv

Policy gradient methods are known to be highly sensitive to the choice of policy parameterization. In particular, the widely used softmax parameterization can induce ill-conditioned optimization landscapes and lead to ex…