EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Speculative decoding accelerates Large Language Model inference through draft-then-verify generation, yet lightweight draft models face coupled efficiency and quality limitations: large-vocabulary output projection is costly, while limited draft capacity and static parameters reduce acceptance under specialized or shifting inputs. Vocabulary pruning lowers projection cost, but static variants miss locally important long-tail tokens, while dynamic variants remain sensitive to preset selection policies and budgets. Moreover, limited draft capacity can leave the draft distribution misaligned even when the target token is covered. Online alignment improves draft quality, but full-parameter updates introduce substantial memory and latency overhead. We introduce EvoSpec, which jointly adapts the active vocabulary and lightweight draft parameters from verification feedback. EvoSpec asynchronously retrieves semantic and statistical token neighbors and performs curriculum-weighted online LoRA alignment while preserving exact target-model verification. On Qwen3-8B/EAGLE-2, EvoSpec reaches a $2.18\times$ speedup over vanilla decoding and a $1.20\times$ gain over EAGLE-2, while improving specialized-domain coverage and using $27\%$ less auxiliary GPU adaptation memory than full-parameter online adaptation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloads and system-level constraints. We present TIDE (Temporal Incremental Draft E…
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autor…
Reinforcement LearningFastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
Group relative policy optimization (GRPO) has demonstrated significant potential in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is im…
Reinforcement LearningMathematical ReasoningSpeculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spe…
Abstractive Text SummarizationMachine TranslationTranslationBASS: Batched Attention-optimized Speculative Sampling
Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating a single sequence. Real-world generativ…
GPUHumanEval