Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

2019-12-14 · Yuwei Luo, Zhuoran Yang, Zhaoran Wang, Mladen Kolar

Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number of agents. We study a fundamental problem of multi-agent linear quadratic regulator (LQR) in a setting where the agents are partially exchangeable. In this setting, we develop a hierarchical actor-critic algorithm, whose computational complexity is independent of the total number of agents, and prove its global linear convergence to the optimal policy. As LQRs are often used to approximate general dynamic systems, this paper provides an important step towards a better understanding of general hierarchical mean-field multi-agent reinforcement learning.

📄 PDF Abstract BibTeX arXiv:1912.06875

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Neural Policy Gradient Methods: Global Optimality and Rates of Convergence

2019-08-29 · ICLR 2020 1 · Lingxiao Wang, Qi Cai, Zhuoran Yang, Zhaoran Wang

Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neur…

Policy Gradient Methods

Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy

2020-08-02 · ICLR 2021 1 · Zuyue Fu, Zhuoran Yang, Zhaoran Wang

We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms. While most existing works on actor-critic employ bi-level or two-timescale upd…

Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning

2024-05-03 · Sihan Zeng, Thinh T. Doan, Justin Romberg

Multi-task reinforcement learning (RL) aims to find a single policy that effectively solves multiple tasks at the same time. This paper presents a constrained formulation for multi-task RL where the goal is to maximize t…

Reinforcement Learning (RL)

Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy

2019-06-25 · Boyi Liu, Qi Cai, Zhuoran Yang, Zhaoran Wang

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy

2019-12-01 · NeurIPS 2019 12 · Boyi Liu, Qi Cai, Zhuoran Yang, Zhaoran Wang

Deep Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)