Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
Speculative decoding is commonly used for reducing the inference latency of large language models. Its effectiveness depends highly on the speculation lookahead (SL)-the number of tokens generated by the draft model at each iteration. In this work we show that the common practice of using the same SL for all iterations (static SL) is suboptimal. We introduce DISCO (DynamIc SpeCulation lookahead Optimization), a novel method for dynamically selecting the SL. Our experiments with four datasets show that DISCO reaches an average speedup of 10% compared to the best static SL baseline, while generating the exact same text.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel. However, this method presents a critical trade-off: it improves throughput in low-load, memory-bound systems but degrades perform…
TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding
Speculative decoding accelerates LLMs by using a lightweight draft model to generate tokens autoregressively before verifying them in parallel with a larger target model. However, determining the optimal number of tokens…
Multi-Armed BanditsDSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
Speculative decoding accelerates large language model inference, but its reliance on a fixed speculation length is suboptimal in large-batch serving environments with diverse requests. This paper explores a new direction…
Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding
Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length i…
Language ModelingLanguage ModellingLarge Language ModelExperience-Driven Dynamic Exits for LLMs with Reinforcement Learning
Large Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation length…
Reinforcement Learning