Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning

2021-10-23 · Sergio Valcarcel Macua, Ian Davies, Aleksi Tukiainen, Enrique Munoz de Cote

We propose a fully distributed actor-critic architecture, named Diff-DAC, with application to multitask reinforcement learning (MRL). During the learning process, agents communicate their value and policy parameters to their neighbours, diffusing the information across a network of agents with no need for a central station. Each agent can only access data from its local task, but aims to learn a common policy that performs well for the whole set of tasks. The architecture is scalable, since the computational and communication cost per agent depends on the number of neighbours rather than the overall number of agents. We derive Diff-DAC from duality theory and provide novel insights into the actor-critic framework, showing that it is actually an instance of the dual ascent method. We prove almost sure convergence of Diff-DAC to a common policy under general assumptions that hold even for deep-neural network approximations. For more restrictive assumptions, we also prove that this common policy is a stationary point of an approximation of the original problem. Numerical results on multitask extensions of common continuous control benchmarks demonstrate that Diff-DAC stabilises learning and has a regularising effect that induces higher performance and better generalisation properties than previous architectures.

📄 PDF Abstract BibTeX arXiv:2110.12306

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning

2017-10-28 · Sergio Valcarcel Macua, Aleksi Tukiainen, Daniel García-Ocaña Hernández, David Baldazo 외

We propose a fully distributed actor-critic algorithm approximated by deep neural networks, named \textit{Diff-DAC}, with application to single-task and to average multitask reinforcement learning (MRL). Each agent has a…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hierarchy Through Composition with Multitask LMDPs

2017-08-01 · ICML 2017 8 · Andrew M. Saxe, Adam C. Earle, Benjamin Rosman

Hierarchical architectures are critical to the scalability of reinforcement learning methods. Most current hierarchical frameworks execute actions serially, with macro-actions comprising sequences of primitive actio…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

2026-06-04 · Johan Obando-Ceron, Lu Li, Scott Fujimoto, Pierre-Luc Bacon 외 arxiv

Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, mak…

Representation Learning Reinforcement Learning Continuous Control

ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning

2022-12-02 · Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim 외

We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion. It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the n…

Robust Multitask Diffusion Normalized M-estimate Subband Adaptive Filtering Algorithm Over Adaptive Networks

2022-10-20 · Wenjing Xu, Haiquan Zhao, Shaohui Lv

In recent years, the multitask diffusion least mean square (MD-LMS) algorithm has been extensively applied in the distributed parameter estimation and target tracking of multitask network. However, its performance is mai…

parameter estimation