Optimizer Fusion: Efficient Training with Better Locality and Parallelism

2021-04-01 · Zixuan Jiang, Jiaqi Gu, Mingjie Liu, Keren Zhu, David Z. Pan

Machine learning frameworks adopt iterative optimizers to train neural networks. Conventional eager execution separates the updating of trainable parameters from forward and backward computations. However, this approach introduces nontrivial training time overhead due to the lack of data locality and computation parallelism. In this work, we propose to fuse the optimizer with forward or backward computation to better leverage locality and parallelism during training. By reordering the forward computation, gradient calculation, and parameter updating, our proposed method improves the efficiency of iterative optimizers. Experimental results demonstrate that we can achieve an up to 20% training time reduction on various configurations. Since our methods do not alter the optimizer algorithm, they can be used as a general "plug-in" technique to the training process.

📄 PDF Abstract BibTeX arXiv:2104.00237

Code (0)

등록된 구현이 없습니다.

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator

2024-11-10 · Kazuki Fujii, Kohei Watanabe, Rio Yokota

In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence Parallelism (SP) and Context Parallelis…

GPU Language Modeling Language Modelling Large Language Model

Demystifying Pipeline Parallelism: First Theory for PipeDream

2026-06-02 · Ivan Ilin, Peter Richtárik arxiv

Training modern machine learning models increasingly requires computation to be distributed across many accelerators. Data parallelism remains the default choice and is often paired with tensor-parallel sharding, but mod…

CADS: Core-Aware Dynamic Scheduler for Multicore Memory Controllers

2019-07-17 · Eduardo Olmedo Sanchez, Xian-He Sun

Memory controller scheduling is crucial in multicore processors, where DRAM bandwidth is shared. Since increased number of requests from multiple cores of processors becomes a source of bottleneck, scheduling the request…

Fairness Reinforcement Learning Reinforcement Learning (RL)Scheduling

d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation

2026-01-12 · Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang 외 arxiv

Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as…

One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

2026-06-29 · Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev 외 arxiv

Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parall…