Central Limit Theorems for Asynchronous Averaged Q-Learning
This paper establishes central limit theorems for Polyak-Ruppert averaged Q-learning under asynchronous updates. We prove a non-asymptotic central limit theorem, where the convergence rate in Wasserstein distance explicitly reflects the dependence on the number of iterations, state-action space size, the discount factor, and the quality of exploration. In addition, we derive a functional central limit theorem, showing that the partial-sum process converges weakly to a Brownian motion.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning
In this paper, we study how to perform statistical inference for quantile temporal difference learning (QTD) in distributional reinforcement learning. Assuming access to a generative model, we first establish functional …
Reinforcement LearningAsymptotic Error Analysis of Multilevel Stochastic Approximations for the Value-at-Risk and Expected Shortfall
Cr\'epey, Frikha, and Louzi (2023) introduced a nested stochastic approximation algorithm and its multilevel acceleration to compute the value-at-risk and expected shortfall of a random financial loss. We hereby establis…
Gaussian Approximation for Asynchronous Q-learning
In this paper, we derive rates of convergence in the high-dimensional central limit theorem for Polyak-Ruppert averaged iterates generated by the asynchronous Q-learning algorithm with a polynomial stepsize $k^{-ω},\, ω\…
Partial Exponential Stability Analysis of Slow-fast Systems via Periodic Averaging
This paper presents some new criteria for partial exponential stability of a slow-fast nonlinear system with a fast scalar variable using periodic averaging methods. Unlike classical averaging techniques, we construct an…
Performance Bounds for Passive Sensing in Asynchronous ISAC Systems -- Appendices
This document contains the appendices for our paper titled ``Performance Bounds for Passive Sensing in Asynchronous ISAC Systems." The appendices include rigorous derivations of key formulas, detailed proofs of the theor…
ISAC