홈 › Papers

FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference

2025-05-28 · Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim

The size and compute characteristics of modern large language models have led to an increased interest in developing specialized kernels tailored for training and inference. Existing kernels primarily optimize for compute utilization, targeting the large-batch training and inference settings. However, low-batch inference, where memory bandwidth and kernel launch overheads contribute are significant factors, remains important for many applications of interest such as in edge deployment and latency-sensitive applications. This paper describes FlashFormer, a proof-of-concept kernel for accelerating single-batch inference for transformer-based large language models. Across various model sizes and quantizations settings, we observe nontrivial speedups compared to existing state-of-the-art inference kernels.

📄 PDF Abstract BibTeX arXiv:2505.22758

Code (0)

등록된 구현이 없습니다.

LLM-42: Enabling Determinism in LLM Inference with Verified Speculation

2026-01-25 · Raja Gond, Aditya K Kamath, Ramachandran Ramjee, Ashish Panwar arxiv

In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels w…

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

2024-10-23 · Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter 외

Each request in LLM inference goes through two phases: compute-bound prefill and memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid batching that combines the prefill and decode phases o…

GPU

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

2024-02-15 · Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim 외

We introduce QUICK, a group of novel optimized CUDA kernels for the efficient inference of quantized Large Language Models (LLMs). QUICK addresses the shared memory bank-conflict problem of state-of-the-art mixed precisi…

GPU Quantization

Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch

2025-11-21 · Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu 외 arxiv

Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and Reinforcement Learning (RL). However, existing LLM serving framewor…

Reinforcement Learning

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

2024-08-21 · Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler 외

As inference on Large Language Models (LLMs) emerges as an important workload in machine learning applications, weight quantization has become a standard technique for efficient GPU deployment. Quantization not only redu…

GPU Quantization Scheduling