Threshold-Based Exclusive Batching for LLM Inference

2026-05-30 · Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma, Shining Wu arxiv

Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

📄 PDF Abstract BibTeX arXiv:2606.00516

Code (0)

등록된 구현이 없습니다.

SynerDiff: Synergetic Continuous Batching for Fast and Parallel Diffusion Model Inference

2026-05-09 · Ziqi Zhou, Peng Yang, Yuxin Liang, Mingliu Liu 외 arxiv

The expansion of Artificial Intelligence-generated content service requires diffusion model serving to simultaneously achieve high throughput and low task end-to-end (E2E) latency. However, existing continuous batching m…

LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference

2020-10-25 · Yujeong Choi, Yunseong Kim, Minsoo Rhu

In cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing m…

BIG-bench Machine Learning Scheduling

Quantized Neural Network Inference with Precision Batching

2020-02-26 · Maximilian Lam, Zachary Yedidia, Colby Banbury, Vijay Janapa Reddi

We present PrecisionBatching, a quantized inference algorithm for speeding up neural network execution on traditional hardware platforms at low bitwidths without the need for retraining or recalibration. PrecisionBatchin…

GPU Language Modeling Language Modelling Natural Language Inference+1

MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching

2025-03-12 · Tairan Xu, Leyang Xue, Zhan Lu, Adrian Jackson 외

This paper presents MoE-Gen, a high-throughput MoE inference system optimized for single-GPU execution. Existing inference systems rely on model-based or continuous batching strategies, originally designed for interactiv…

GPU

Green MLOps: Closed-Loop, Energy-Aware Inference with NVIDIA Triton, FastAPI, and Bio-Inspired Thresholding

2026-01-06 · Mustapha Hamdi, Mourad Jabou arxiv

Energy efficiency is a first-order concern in AI deployment, as long-running inference can exceed training in cumulative carbon impact. We propose a bio-inspired framework that maps protein-folding energy basins to infer…