SpecMD: A Comprehensive Study On Speculative Expert Prefetching

2026-02-03 · Duc Hoang, Ajay Jaiswal, Mohammad Samragh, Minsik Cho arxiv

Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.

📄 PDF Abstract BibTeX arXiv:2602.03921

Code (0)

등록된 구현이 없습니다.

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios

2026-02-12 · Shuhuai Li, Jianghao Lin, Dongdong Ge, Yinyu Ye arxiv

Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices. Existing offloading strategies struggle with I/O bottlenecks due to the dynamic, low-information nature of a…

MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts

2025-11-18 · Wenfeng Wang, Jiacheng Liu, Xiaofeng Hou, Xinfeng Xia 외 arxiv

The immense memory requirements of state-of-the-art Mixture-of-Experts (MoE) models present a significant challenge for inference, often exceeding the capacity of a single accelerator. While offloading experts to host me…

A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference

2026-06-13 · Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth 외 arxiv

Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost. By …

Code Generation

Speculating Experts Accelerates Inference for Mixture-of-Experts

2026-03-09 · Vivan Madan, Prajwal Singhania, Abhinav Bhatele, Tom Goldstein 외 arxiv

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained…

NVR: Vector Runahead on NPUs for Sparse Memory Access

2025-02-19 · Hui Wang, Zhengpeng Zhao, Jing Wang, Yushu Du 외

Deep Neural Networks are increasingly leveraging sparsity to reduce the scaling up of model parameter size. However, reducing wall-clock time through sparsity and pruning remains challenging due to irregular memory acces…