Predictable Interval MDPs through Entropy Regularization

2024-03-25 · Menno van Zutphen, Giannis Delimpaltadakis, Maurice Heemels, Duarte Antunes

Regularization of control policies using entropy can be instrumental in adjusting predictability of real-world systems. Applications benefiting from such approaches range from, e.g., cybersecurity, which aims at maximal unpredictability, to human-robot interaction, where predictable behavior is highly desirable. In this paper, we consider entropy regularization for interval Markov decision processes (IMDPs). IMDPs are uncertain MDPs, where transition probabilities are only known to belong to intervals. Lately, IMDPs have gained significant popularity in the context of abstracting stochastic systems for control design. In this work, we address robust minimization of the linear combination of entropy and a standard cumulative cost in IMDPs, thereby establishing a trade-off between optimality and predictability. We show that optimal deterministic policies exist, and devise a value-iteration algorithm to compute them. The algorithm solves a number of convex programs at each step. Finally, through an illustrative example we show the benefits of penalizing entropy in IMDPs.

📄 PDF Abstract BibTeX arXiv:2403.16711

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization

2021-10-17 · Donghao Ying, Yuhao Ding, Javad Lavaei

We study entropy-regularized constrained Markov decision processes (CMDPs) under the soft-max parameterization, in which an agent aims to maximize the entropy-regularized value function while satisfying constraints on th…

High entropy leads to symmetry-equivariant policies in Dec-POMDPs

2025-11-27 · Johannes Forkel, Constantin Ruhdorfer, Michael Beukman, Andreas Bulling 외 arxiv

We prove that in any Dec-POMDP, sufficiently high entropy regularization ensures that the policy gradient flow with tabular softmax parametrization always converges, for any initialization, to the same joint policy, and …

Entropy-Regularized Partially Observed Markov Decision Processes

2021-12-22 · Timothy L. Molloy, Girish N. Nair

We investigate partially observed Markov decision processes (POMDPs) with cost functions regularized by entropy terms describing state, observation, and control uncertainty. Standard POMDP techniques are shown to offer b…

State Estimation

Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning

2017-09-19 · Kyungjae Lee, Sungjoon Choi, Songhwai Oh

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A unified view of entropy-regularized Markov decision processes

2017-05-22 · Gergely Neu, Anders Jonsson, Vicenç Gómez

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimizati…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)