Gradient Coding with Clustering and Multi-message Communication

2019-03-05 · Emre Ozfatura, Deniz Gunduz, Sennur Ulukus

Gradient descent (GD) methods are commonly employed in machine learning problems to optimize the parameters of the model in an iterative fashion. For problems with massive datasets, computations are distributed to many parallel computing servers (i.e., workers) to speed up GD iterations. While distributed computing can increase the computation speed significantly, the per-iteration completion time is limited by the slowest straggling workers. Coded distributed computing can mitigate straggling workers by introducing redundant computations; however, existing coded computing schemes are mainly designed against persistent stragglers, and partial computations at straggling workers are discarded, leading to wasted computational capacity. In this paper, we propose a novel gradient coding (GC) scheme which allows multiple coded computations to be conveyed from each worker to the master per iteration. We numerically show that the proposed GC with multi-message communication (MMC) together with clustering provides significant improvements in the average completion time (of each iteration), with minimal or no increase in the communication load.

📄 PDF Abstract BibTeX arXiv:1903.01974

Code (0)

등록된 구현이 없습니다.

Tasks

ClusteringDistributed Computing

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning

2023-08-07 · Jingdi Chen, Tian Lan, Carlee Joe-Wong

Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in partially observable Markov Decision Processes. Existing works often rely on black-box methods to encode local information/feat…

Clustering Multi-agent Reinforcement Learning Online Clustering reinforcement-learning

Scalability of Message Encoding Techniques for Continuous Communication Learned with Multi-Agent Reinforcement Learning

2023-08-09 · Astrid Vanneste, Thomas Somers, Simon Vanneste, Kevin Mets 외

Many multi-agent systems require inter-agent communication to properly achieve their goal. By learning the communication protocol alongside the action protocol using multi-agent reinforcement learning techniques, the age…

Multi-agent Reinforcement Learning

Low Entropy Communication in Multi-Agent Reinforcement Learning

2023-02-10 · Lebin Yu, Yunbo Qiu, Qiexiang Wang, Xudong Zhang 외

Communication in multi-agent reinforcement learning has been drawing attention recently for its significant role in cooperation. However, multi-agent systems may suffer from limitations on communication resources and thu…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Byzantine-Robust and Communication-Efficient Distributed Training: Compressive and Cyclic Gradient Coding

2026-03-17 · Chengxi Li, Youssef Allouah, Rachid Guerraoui, Mikael Skoglund 외 arxiv

In this paper, we study the problem of distributed training (DT) under Byzantine attacks with communication constraints. While prior work has developed various robust aggregation rules at the server to enhance robustness…

Gradient Coding with Dynamic Clustering for Straggler Mitigation

2020-11-03 · Baturalp Buyukates, Emre Ozfatura, Sennur Ulukus, Deniz Gunduz

In distributed synchronous gradient descent (GD) the main performance bottleneck for the per-iteration completion time is the slowest \textit{straggling} workers. To speed up GD iterations in the presence of stragglers, …

Clustering