Performance of NPG in Countable State-Space Average-Cost RL
We consider policy optimization methods in reinforcement learning settings where the state space is arbitrarily large, or even countably infinite. The motivation arises from control problems in communication networks, matching markets, and other queueing systems. We consider Natural Policy Gradient (NPG), which is a popular algorithm for finite state spaces. Under reasonable assumptions, we derive a performance bound for NPG that is independent of the size of the state space, provided the error in policy evaluation is within a factor of the true value function. We obtain this result by establishing new policy-independent bounds on the solution to Poisson's equation, i.e., the relative value function, and by combining these bounds with previously known connections between MDPs and learning from experts.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Robust utility maximisation under proportional transaction costs for càdlàg price processes
We consider robust utility maximisation in continuous-time financial markets with proportional transaction costs under model uncertainty. For this purpose, we work in the framework of Chau and R\'asonyi (2019), where rob…
Open-Ended Question AnsweringThiele's Differential Equation Based on Markov Jump Processes with Non-countable State Space
In modern life insurance, Markov processes in continuous time on a finite or at least countable state space have been over the years an important tool for the modelling of the states of an insured. Motivated by applicati…
Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space
Models of many real-life applications, such as queuing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce…
Thompson SamplingBayesian Optimization with a Finite Budget: An Approximate Dynamic Programming Approach
We consider the problem of optimizing an expensive objective function when a finite budget of total evaluations is prescribed. In that context, the optimal solution strategy for Bayesian optimization can be formulated as…
Bayesian OptimizationIV-GNN : Interval Valued Data Handling Using Graph Neural Network
Graph Neural Network (GNN) is a powerful tool to perform standard machine learning on graphs. To have a Euclidean representation of every node in the Non-Euclidean graph-like data, GNN follows neighbourhood aggregation a…
Graph ClassificationGraph Neural Network