Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

2021-07-26 · Gangshan Jing, He Bai, Jemin George, Aranya Chakrabortty, Piyush K. Sharma

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

📄 PDF Abstract BibTeX arXiv:2107.12416

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning (RL)

Zeroth-order Asynchronous Doubly Stochastic Algorithm with Variance Reduction

2016-12-05 · Bin Gu, Zhouyuan Huo, Heng Huang

Zeroth-order (derivative-free) optimization attracts a lot of attention in machine learning, because explicit gradient calculations may be computationally expensive or infeasible. To handle large scale problems both in v…

Faster Derivative-Free Stochastic Algorithm for Shared Memory Machines

2018-07-01 · ICML 2018 7 · Bin Gu, Zhouyuan Huo, Cheng Deng, Heng Huang

Asynchronous parallel stochastic gradient optimization has been playing a pivotal role to solve large-scale machine learning problems in big data applications. Zeroth-order (derivative-free) methods estimate the gra…

Ensemble Learning

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

2020-06-18 · Yan Zhang, Michael M. Zavlanos

In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Asynchronous Message-Passing and Zeroth-Order Optimization Based Distributed Learning with a Use-Case in Resource Allocation in Communication Networks

2023-11-08 · Pourya Behmandpoor, Marc Moonen, Panagiotis Patrinos

Distributed learning and adaptation have received significant interest and found wide-ranging applications in machine learning and signal processing. While various approaches, such as shared-memory optimization, multi-ta…

Distributed Optimization Federated Learning Multi-Task Learning

Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

2021-09-29 · Deming Yuan, Lei Wang, Alexandre Proutiere, Guodong Shi

Zeroth-order optimization has become increasingly important in complex optimization and machine learning when cost functions are impossible to be described in closed analytical forms. The key idea of zeroth-order o…