Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments
As a primary contribution, we present a convergence theorem for stochastic iterations, and in particular, Q-learning iterates, under a general, possibly non-Markovian, stochastic environment. Our conditions for convergence involve an ergodicity and a positivity criterion. We provide a precise characterization on the limit of the iterates and conditions on the environment and initializations for convergence. As our second contribution, we discuss the implications and applications of this theorem to a variety of stochastic control problems with non-Markovian environments involving (i) quantized approximations of fully observed Markov Decision Processes (MDPs) with continuous spaces (where quantization break down the Markovian structure), (ii) quantized approximations of belief-MDP reduced partially observable MDPS (POMDPs) with weak Feller continuity and a mild version of filter stability (which requires the knowledge of the model by the controller), (iii) finite window approximations of POMDPs under a uniform controlled filter stability (which does not require the knowledge of the model), and (iv) for multi-agent models where convergence of learning dynamics to a new class of equilibria, subjective Q-learning equilibria, will be studied. In addition to the convergence theorem, some implications of the theorem above are new to the literature and others are interpreted as applications of the convergence theorem. Some open problems are noted.
Code (0)
등록된 구현이 없습니다.
Tasks
Q-LearningQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Principled Learning-to-Communicate with Quasi-Classical Information Structures
Learning-to-communicate (LTC) in partially observable environments has received increasing attention in deep multi-agent reinforcement learning, where the control and communication strategies are jointly learned. Meanwhi…
Multi-agent Reinforcement LearningCommon Information Approach for Static Team Problems with Polish Spaces and Existence of Optimal Policies
In this paper, we demonstrate the existence of team-optimal strategies for static teams under observation-sharing information structures. Assuming that agents can access shared observations, we begin by converting the te…
A case study comparing both stochastic and worst-case robust control co-design under different control structures
As uncertainty considerations become increasingly important aspects of concurrent plant and control optimization, it is imperative to identify and compare the impact of uncertain control co-design (UCCD) formulations on …
Communication and Control Co-design in Non-cooperative Games
In this article, we revisit a communication-control co-design problem for a class of two-player stochastic differential games on an infinite horizon. Each 'player' represents two active decision makers, namely a schedule…
Learning to Schedule Learning rate with Graph Neural Networks
Recent decades have witnessed great development of stochastic optimization in training deep neural networks. Learning rate scheduling is one of the most important factors that influence the performance of stochastic opti…
Benchmarkingimage-classificationImage ClassificationScheduling+1