Markov Decision Processes under External Temporal Processes

2023-05-25 · Ranga Shaarad Ayyagari, Ambedkar Dukkipati

Most reinforcement learning algorithms treat the context under which they operate as a stationary, isolated, and undisturbed environment. However, in real world applications, environments constantly change due to a variety of external events. To address this problem, we study Markov Decision Processes (MDP) under the influence of an external temporal process. First, we formalize this notion and derive conditions under which the problem becomes tractable with suitable solutions. We propose a policy iteration algorithm to solve this problem and theoretically analyze its performance. Our analysis addresses the non-stationarity present in the MDP as a result of non-Markovian events, necessitating the formulation of policies that are contingent upon both the current state and a history of prior events. Additionally, we derive insights regarding the sample complexity of the algorithm and incorporate factors that define the exogenous temporal process into the established bounds. Finally, we perform experiments to demonstrate our findings within a traditional control environment.

📄 PDF Abstract BibTeX arXiv:2305.16056

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning

Regular Decision Processes for Grid Worlds

2021-11-05 · Nicky Lenaers, Martijn van Otterlo

Markov decision processes are typically used for sequential decision making under uncertainty. For many aspects however, ranging from constrained or safe specifications to various kinds of temporal (non-Markovian) depend…

Decision Making Decision Making Under Uncertainty Incremental Learning Sequential Decision Making

Model-free Reinforcement Learning for Branching Markov Decision Processes

2021-06-12 · Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi 외

We study reinforcement learning for the optimal control of Branching Markov Decision Processes (BMDPs), a natural extension of (multitype) Branching Markov Chains (BMCs). The state of a (discrete-time) BMCs is a collecti…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Solution to Time-Varying Markov Decision Processes

2016-05-03 · Lantao Liu, Gaurav S. Sukhatme

We consider a decision-making problem where the environment varies both in space and time. Such problems arise naturally when considering e.g., the navigation of an underwater robot amidst ocean currents or the navigatio…

Decision Making

A Tutorial Introduction to Reinforcement Learning

2023-04-03 · Mathukumalli Vidyasagar

In this paper, we present a brief survey of Reinforcement Learning (RL), with particular emphasis on Stochastic Approximation (SA) as a unifying theme. The scope of the paper includes Markov Reward Processes, Markov Deci…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Online Markov Decision Processes with Non-oblivious Strategic Adversary

2021-10-07 · Le Cong Dinh, David Henry Mguni, Long Tran-Thanh, Jun Wang 외

We study a novel setting in Online Markov Decision Processes (OMDPs) where the loss function is chosen by a non-oblivious strategic adversary who follows a no-external regret algorithm. In this setting, we first demonstr…