SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models

2025-02-28 · Han-Byul Kim, Duc Hoang, Arnav Kundu, Mohammad Samragh, Minsik Cho

With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. However, communication overheads from popular distributed inference techniques such as Tensor Parallelism pose a significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD), to reduce communication overheads in tensor parallelism by selectively dropping synchronization on attention outputs. In detail, we first propose a block design that allows execution to proceed without communication through SPD. Second, we apply different SPD strategies to attention blocks based on their sensitivity to the model accuracy. The proposed methods effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference, offering a scalable solution for diverse distributed environments: SPD offered about 20% overall inference latency reduction with < 1% accuracy regression for LLaMA2-70B inference over 8 GPUs.

📄 PDF Abstract BibTeX arXiv:2502.20727

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization

2026-02-07 · Chong Wang, Nan Du, Tom Gunter, Tao Lei 외 arxiv

Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conv…

Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism

2026-06-08 · Sergei Vorobyov, Eugene Ilyushin arxiv

Formal neural network verification -- proving that a network satisfies safety properties for *all* inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algor…

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

2025-04-21 · Dennis Liu, Zijie Yan, Xin Yao, Tong Liu 외

Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageable computation costs. However, efficient…

Mixture-of-Experts

Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems

2025-06-21 · Haowei Yang, Yu Tian, Zhongheng Yang, Zhao Wang 외

With the rapid adoption of large language models (LLMs) in recommendation systems, the computational and communication bottlenecks caused by their massive parameter sizes and large data volumes have become increasingly p…

Language Modeling Language Modelling Large Language Model model+2

Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel

2026-04-14 · Hongyi Jin, Bohan Hou, Guanjie Wang, Ruihang Lai 외 arxiv

Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operat…