A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs
During the deployment of Large Language Models (LLMs), the autoregressive decoding phase on heterogeneous NPU platforms (e.g., Ascend 910B) faces severe memory-bound challenges. This study reveals the ``Model Scaling Paradox'' caused by the static deployment of single-sized models. It also points out the kernel synchronization overhead of fine-grained speculative decoding \cite{leviathan2023fast, chen2023speculative} under NPU computational graph compilation, and the severe limitations of purely relying on micro-level acceleration algorithms like Prompt LookUp Decoding (PLD)
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
Deploying large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution, yet no prior study has systematically characterized NPU effectiveness at the operator and pipeline level. We presen…
FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs
FlashAttention series has been widely applied in the inference of large language models (LLMs). However, FlashAttention series only supports the high-level GPU architectures, e.g., Ampere and Hopper. At present, FlashAtt…
CPUGPUContext-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units
The proliferation of large language models has driven demand for long-context inference on resource-constrained edge platforms. However, deploying these models on Neural Processing Units (NPUs) presents significant chall…
Causal InferenceANVIL: Accelerator-Native Video Interpolation via Codec Motion Vector Priors
Real-time 30-to-60 fps video frame interpolation on mobile neural processing units (NPUs) requires each synthesized frame within 33.3 ms. We show that mainstream flow-based video frame interpolation faces three structura…
Video Frame InterpolationCharacterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment
Vision-Language-Action (VLA) models are promising for generalist robot control, but on-robot deployment is bottlenecked by real-time inference under tight cost and energy budgets. Most prior evaluations rely on desktop-g…