Highly Available Data Parallel ML training on Mesh Networks

2020-11-06 · Sameer Kumar, Norm Jouppi

Data parallel ML models can take several days or weeks to train on several accelerators. The long duration of training relies on the cluster of resources to be available for the job to keep running for the entire duration. On a mesh network this is challenging because failures will create holes in the mesh. Packets must be routed around the failed chips for full connectivity. In this paper, we present techniques to route gradient summation allreduce traffic around failed chips on 2-D meshes. We evaluate performance of our fault tolerant allreduce techniques via the MLPerf-v0.7 ResNet-50 and BERT benchmarks. Performance results show minimal impact to training throughput on 512 and 1024 TPU-v3 chips.

📄 PDF Abstract BibTeX arXiv:2011.03605

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Residual Connection 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Mesh-TensorFlow: Deep Learning for Supercomputers

2018-11-05 · NeurIPS 2018 12 · Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran 외

Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program-Multiple-Data (SPMD) programming. Howev…

Deep Learning Language Modeling Language Modelling

Topology Sculptor, Shape Refiner: Discrete Diffusion Model for High-Fidelity 3D Meshes Generation

2025-10-24 · Kaiyu Song, Hanjiang Lai, Yaqing Zhang, Chuangjian Cai 외 arxiv

In this paper, we introduce Topology Sculptor, Shape Refiner (TSSR), a novel method for generating high-quality, artist-style 3D meshes based on Discrete Diffusion Models (DDMs). Our primary motivation for TSSR is to ach…

On Optimizing the Communication of Model Parallelism

2022-11-10 · Yonghao Zhuang, Hexu Zhao, Lianmin Zheng, Zhuohan Li 외

We study a novel and important communication pattern in large-scale model-parallel deep learning (DL), which we call cross-mesh resharding. This pattern emerges when the two paradigms of model parallelism - intra-operato…

model

Parallelised Differentiable Straightest Geodesics for 3D Meshes

2026-03-16 · Hippolyte Verninas, Caner Korkmaz, Stefanos Zafeiriou, Tolga Birdal 외 arxiv

Machine learning has been progressively generalised to operate within non-Euclidean domains, but geometrically accurate methods for learning on surfaces are still falling behind. The lack of closed-form Riemannian operat…

Dmsh: A Multi-Agent Reinforcement Learning Framework for All-Quad Mesh Generation

2026-06-09 · Anirudh Kalyan, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabczuk 외 arxiv

Generating high-quality meshes for arbitrary geometries remains a fundamental bottleneck in computational engineering, often demanding heuristic tuning and semi-manual workflows. In this paper, we introduce Dmsh, a first…

Multi-agent Reinforcement Learning