Graph-Structured Speculative Decoding

2024-07-23 · Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan

Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models (LLMs) by employing a small language model to draft a hypothesis sequence, which is then validated by the LLM. The effectiveness of this approach heavily relies on the balance between performance and efficiency of the draft model. In our research, we focus on enhancing the proportion of draft tokens that are accepted to the final output by generating multiple hypotheses instead of just one. This allows the LLM more options to choose from and select the longest sequence that meets its standards. Our analysis reveals that hypotheses produced by the draft model share many common token sequences, suggesting a potential for optimizing computation. Leveraging this observation, we introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. This structure enables us to efficiently predict and merge recurring token sequences, vastly reducing the computational demands of the draft model. We term this approach Graph-structured Speculative Decoding (GSD). We apply GSD across a range of LLMs, including a 70-billion parameter LLaMA-2 model, and observe a remarkable speedup of 1.73$\times$ to 1.96$\times$, significantly surpassing standard speculative decoding.

📄 PDF Abstract BibTeX arXiv:2407.16207

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModellingSmall Language Model

Methods 이 논문이 사용한 방법론

Focus 설명 없음

Speculative Streaming: Fast LLM Inference without Auxiliary Models

2024-02-16 · Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason 외

Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often invol…

Language Modeling Language Modelling

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

2026-03-09 · Chang Han, Yijie Hu, Jingling Liu arxiv

Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens p…

Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding

2025-09-15 · Mingxiao Huo, Jiayi Zhang, Hewei Wang, Jinfeng Xu 외 arxiv

Vision-Language Models (VLMs) enable powerful multimodal reasoning but suffer from slow autoregressive inference, limiting their deployment in real-time applications. We introduce Spec-LLaVA, a system that applies specul…

Multimodal Reasoning

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

2026-05-31 · Xin Su, Dawid Majchrowski, Fangyuan Yu, Vanshil Atul Shah 외 arxiv

Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with …

Decoding Speculative Decoding

2024-02-02 · Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman

Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate s…