STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining
Natural Language Processing (NLP) inference is seeing increasing adoption by mobile applications, where on-device inference is desirable for crucially preserving user data privacy and avoiding network roundtrips. Yet, the unprecedented size of an NLP model stresses both latency and memory, creating a tension between the two key resources of a mobile device. To meet a target latency, holding the whole model in memory launches execution as soon as possible but increases one app's memory footprints by several times, limiting its benefits to only a few inferences before being recycled by mobile memory management. On the other hand, loading the model from storage on demand incurs IO as long as a few seconds, far exceeding the delay range satisfying to a user; pipelining layerwise model loading and execution does not hide IO either, due to the high skewness between IO and computation delays. To this end, we propose Speedy Transformer Inference (STI). Built on the key idea of maximizing IO/compute resource utilization on the most important parts of a model, STI reconciles the latency v.s. memory tension via two novel techniques. First, model sharding. STI manages model parameters as independently tunable shards, and profiles their importance to accuracy. Second, elastic pipeline planning with a preload buffer. STI instantiates an IO/compute pipeline and uses a small buffer for preload shards to bootstrap execution without stalling at early stages; it judiciously selects, tunes, and assembles shards per their importance for resource-elastic execution, maximizing inference accuracy. Atop two commodity SoCs, we build STI and evaluate it against a wide range of NLP tasks, under a practical range of target latencies, and on both CPU and GPU. We demonstrate that STI delivers high accuracies with 1-2 orders of magnitude lower memory, outperforming competitive baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
CPUGPUManagementMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AntBatchInfer: Elastic Batch Inference in the Kubernetes Cluster
Offline batch inference is a common task in the industry for deep learning applications, but it can be challenging to ensure stability and performance when dealing with large amounts of data and complicated inference pip…
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
The size of Transformer models is growing at an unprecedented pace. It has only taken less than one year to reach trillion-level parameters after the release of GPT-3 (175B). Training such models requires both substantia…
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
We propose Joint-DetNAS, a unified NAS framework for object detection, which integrates 3 key components: Neural Architecture Search, pruning, and Knowledge Distillation. Instead of naively pipelining these techniques, o…
Knowledge DistillationNeural Architecture Searchobject-detectionObject DetectionDesign and Selection of Additional Residuals to Enhance Fault Isolation of a Turbocharged Spark Ignited Engine System
This paper presents a method to enhance fault isolation without adding physical sensors on a turbocharged spark ignited petrol engine system by designing additional residuals from an initial observer-based residuals setu…
The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution
While ML model training and inference are both GPU-intensive, CPU-based data processing is often the bottleneck. Distributed data processing systems based on the batch or stream processing models assume homogeneous resou…
CPUGPU