Equilibrium in Misspecified Markov Decision Processes
We study Markov decision problems where the agent does not know the transition probability function mapping current states and actions to future states. The agent has a prior belief over a set of possible transition functions and updates beliefs using Bayes' rule. We allow her to be misspecified in the sense that the true transition probability function is not in the support of her prior. This problem is relevant in many economic settings but is usually not amenable to analysis by the researcher. We make the problem tractable by studying asymptotic behavior. We propose an equilibrium notion and provide conditions under which it characterizes steady state behavior. In the special case where the problem is static, equilibrium coincides with the single-agent version of Berk-Nash equilibrium (Esponda and Pouzo (2016)). We also discuss subtle issues that arise exclusively in dynamic settings due to the possibility of a negative value of experimentation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On Existence of Berk-Nash Equilibria in Misspecified Markov Decision Processes with Infinite Spaces
Model misspecification is a critical issue in many areas of theoretical and empirical economics. In the specific context of misspecified Markov Decision Processes, Esponda and Pouzo (2021) defined the notion of Berk-Nash…
Iterative Hierarchical Optimization for Misspecified Problems (IHOMP)
For complex, high-dimensional Markov Decision Processes (MDPs), it may be necessary to represent the policy with function approximation. A problem is misspecified whenever, the representation cannot express any policy wi…
Robust $Q$-learning Algorithm for Markov Decision Processes under Wasserstein Uncertainty
We present a novel $Q$-learning algorithm tailored to solve distributionally robust Markov decision problems where the corresponding ambiguity set of transition probabilities for the underlying Markov decision process is…
Q-LearningBad Values but Good Behavior: Learning Highly Misspecified Bandits and MDPs
Parametric, feature-based reward models are employed by a variety of algorithms in decision-making settings such as bandits and Markov decision processes (MDPs). The typical assumption under which the algorithms are anal…
Decision MakingMulti-Armed BanditsQ-LearningApproximate State Abstraction for Markov Games
This paper introduces state abstraction for two-player zero-sum Markov games (TZMGs), where the payoffs for the two players are determined by the state representing the environment and their respective actions, with stat…