Imprecise Markov Semigroups and their Ergodicity

2024-04-30 · Michele Caprio

We introduce the concept of an imprecise Markov semigroup $\mathbf{Q}$. It is a tool that allows to represent ambiguity around both the initial and the transition probabilities of a Markov process via a compact collection of Markov semigroups, each associated with a (possibly different) Markov process. We use techniques from set theory, topology, geometry, and probability to study the ergodic behavior of $\mathbf{Q}$. We show that, if the initial distribution of the Markov processes associated with the elements of $\mathbf{Q}$ is known and invariant, under some conditions that also involve the geometry of the state space, eventually the ambiguity around their transition probability fades. We call this property ergodicity of the imprecise Markov semigroup, and we relate it to the classical notion of ergodicity. We prove ergodicity both when the state space is Euclidean or a Riemannian manifold, and when it is an arbitrary measurable space. The importance of our findings for the fields of machine learning and computer vision is also discussed.

📄 PDF Abstract BibTeX arXiv:2405.00081

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Adam-like Algorithm with Smooth Clipping Attains Global Minima: Analysis Based on Ergodicity of Functional SDEs

2023-11-29 · Keisuke Suzuki

In this paper, we prove that an Adam-type algorithm with smooth clipping approaches the global minimizer of the regularized non-convex loss function. Adding smooth clipping and taking the state space as the set of all tr…

Reinforcement Learning with Function Approximation for Non-Markov Processes

2026-01-01 · Ali Devran Kara arxiv

We study reinforcement learning methods with linear function approximation under non-Markov state and cost processes. We first consider the policy evaluation method and show that the algorithm converges under suitable er…

Reinforcement Learning

On the Geometric Ergodicity of Hamiltonian Monte Carlo

2016-01-29 · Samuel Livingstone, Michael Betancourt, Simon Byrne, Mark Girolami

We establish general conditions under which Markov chains produced by the Hamiltonian Monte Carlo method will and will not be geometrically ergodic. We consider implementations with both position-independent and position…

Position

Approximating Euclidean by Imprecise Markov Decision Processes

2020-06-26 · Manfred Jaeger, Giorgio Bacci, Giovanni Bacci, Kim Guldstrand Larsen 외

Euclidean Markov decision processes are a powerful tool for modeling control problems under uncertainty over continuous domains. Finite state imprecise, Markov decision processes can be used to approximate the behavior o…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Safe Exploration in Markov Decision Processes

2012-05-22 · Teodor Mihai Moldovan, Pieter Abbeel

In environments with uncertain dynamics exploration is necessary to learn how to perform well. Existing reinforcement learning algorithms provide strong exploration guarantees, but they tend to rely on an ergodicity assu…

Safe Exploration