Soft BIBD and Product Gradient Codes
Gradient coding is a coding theoretic framework to provide robustness against slow or unresponsive machines, known as stragglers, in distributed machine learning applications. Recently, Kadhe et al. proposed a gradient code based on a combinatorial design, called balanced incomplete block design (BIBD), which is shown to outperform many existing gradient codes in worst-case adversarial straggling scenarios. However, parameters for which such BIBD constructions exist are very limited. In this paper, we aim to overcome such limitations and construct gradient codes which exist for a wide range of system parameters while retaining the superior performance of BIBD gradient codes. Two such constructions are proposed, one based on a probabilistic construction that relax the stringent BIBD gradient code constraints, and the other based on taking the Kronecker product of existing gradient codes. The proposed gradient codes allow flexible choices of system parameters while retaining comparable error performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Gradient Coding Based on Block Designs for Mitigating Adversarial Stragglers
Distributed implementations of gradient-based methods, wherein a server distributes gradient computations across worker machines, suffer from slow running machines, called 'stragglers'. Gradient coding is a coding-theore…
From Signal Space To STP-CS
Under the assumption that a finite signal with different sampling lengths or different sampling frequencies is considered as equivalent, the signal space is considered as the quotient space of $\mathbb{R}^{\infty}$ over …
compressed sensing$hv$-Block Cross Validation is not a BIBD: a Note on the Paper by Jeff Racine (2000)
This note corrects a mistake in the paper "consistent cross-validatory model-selection for dependent data: $hv$-block cross-validation" by Racine (2000). In his paper, he implied that the therein proposed $hv$-block cros…
Model SelectionOpen-Ended Question AnsweringAlternatives to the Scaled Dot Product for Attention in the Transformer Neural Network Architecture
The transformer neural network architecture uses a form of attention in which the dot product of query and key is divided by the square root of the key dimension before applying softmax. This scaling of the dot product i…
A Unified Coded Deep Neural Network Training Strategy Based on Generalized PolyDot Codes for Matrix Multiplication
This paper has two contributions. First, we propose a novel coded matrix multiplication technique called Generalized PolyDot codes that advances on existing methods for coded matrix multiplication under storage and commu…