Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning
Cooperative multi-agent reinforcement learning is a decentralized paradigm in sequential decision making where agents distributed over a network iteratively collaborate with neighbors to maximize global (network-wide) notions of rewards. Exact computations typically involve a complexity that scales exponentially with the number of agents. To address this curse of dimensionality, we design a scalable algorithm based on the Natural Policy Gradient framework that uses local information and only requires agents to communicate with neighbors within a certain range. Under standard assumptions on the spatial decay of correlations for the transition dynamics of the underlying Markov process and the localized learning policy, we show that our algorithm converges to the globally optimal policy with a dimension-free statistical and computational complexity, incurring a localization error that does not depend on the number of agents and converges to zero exponentially fast as a function of the range of communication.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Sequential Decision MakingSimilar Papers 제목 키워드 기반
Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
We study sequential decision making problems aimed at maximizing the expected total reward while satisfying a constraint on the expected total utility. We employ the natural policy gradient method to solve the discounted…
Decision MakingSequential Decision MakingNatural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
We study sequential decision-making problems in which each agent aims to maximize the expected total reward while satisfying a constraint on the expected total utility. We employ the natural policy gradient method to sol…
Decision MakingSequential Decision MakingReinforcement Learning in Linear Quadratic Deep Structured Teams: Global Convergence of Policy Gradient Methods
In this paper, we study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms for linear quadratic deep structured teams. In such systems, agents are …
Policy Gradient MethodsRecurrent Natural Policy Gradient for POMDPs
In this paper, we study a natural policy gradient method based on recurrent neural networks (RNNs) for partially-observable Markov decision processes, whereby RNNs are used for policy parameterization and policy evaluati…
Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning. Hybrid appro…
Reinforcement LearningContinuous Control