Primitives for Dynamic Big Model Parallelism

2014-06-18 · Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A. Gibson, Eric P. Xing

When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness additional memory and processors. However, naive, unstructured parallelization of ML algorithms can make inefficient use of distributed memory, while failing to obtain proportional convergence speedups - or can even result in divergence. We develop a framework of primitives for dynamic model-parallelism, STRADS, in order to explore partitioning and update scheduling of model variables in distributed ML algorithms - thus improving their memory efficiency while presenting new opportunities to speed up convergence without compromising inference correctness. We demonstrate the efficacy of model-parallel algorithms implemented in STRADS versus popular implementations for Topic Modeling, Matrix Factorization and Lasso.

📄 PDF Abstract BibTeX arXiv:1406.4580

Code (0)

등록된 구현이 없습니다.

Tasks

modelScheduling

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Automap: Towards Ergonomic Automated Parallelism for ML Models

2021-12-06 · Michael Schaarschmidt, Dominik Grewe, Dimitrios Vytiniotis, Adam Paszke 외

The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline parallelism. Implementing these methods …

dMath: A Scalable Linear Algebra and Math Library for Heterogeneous GP-GPU Architectures

2016-04-05 · Steven Eliuk, Cameron Upright, Anthony Skjellum

A new scalable parallel math library, dMath, is presented in this paper that demonstrates leading scaling when using intranode, or internode, hybrid-parallelism for deep-learning. dMath provides easy-to-use distributed b…

GPU Management Math

Multi-Robot Motions in Milliseconds: Vector-Accelerated Primitives for Sampling-Based Planning

2026-04-27 · James D. Motes, Marco Morales, Nancy M. Amato arxiv

In this paper, we extend the recent Vector-Accelerated Motion Planning (VAMP) framework to multi-robot motion planning (MRMP). We develop two vector-accelerated primitives, multi-robot MotionValidation (MotVal) and FindF…

Motion Planning

A Linear Algebraic Approach to Model Parallelism in Deep Learning

2020-06-04 · Russell J. Hewett, Thomas J. Grady II

Training deep neural networks (DNNs) in large-cluster computing environments is increasingly necessary, as networks grow in size and complexity. Local memory and processing limitations require robust data and model paral…

Deep Learning

RLlib: Abstractions for Distributed Reinforcement Learning

2017-12-26 · ICML 2018 7 · Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara 외

Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation. We argue for distributing RL component…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)rllib