Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline
Multiple Deep Neural Networks (DNNs) integrated into single Deep Learning (DL) inference pipelines e.g. Multi-Task Learning (MTL) or Ensemble Learning (EL), etc., albeit very accurate, pose challenges for edge deployment. In these systems, models vary in their quantization tolerance and resource demands, requiring meticulous tuning for accuracy-latency balance. This paper introduces an automated heterogeneous quantization approach for DL inference pipelines with multiple DNNs.
Code (0)
등록된 구현이 없습니다.
Tasks
Ensemble LearningMulti-Task LearningQuantizationSimilar Papers 제목 키워드 기반
Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
Although the quest for more accurate solutions is pushing deep learning research towards larger and more complex algorithms, edge devices demand efficient inference and therefore reduction in model size, latency and ener…
QuantizationWhen NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
Deploying large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution, yet no prior study has systematically characterized NPU effectiveness at the operator and pipeline level. We presen…
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design pr…
VibeVoice-ASR-BitNet Technical Report
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the …
MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization
Matryoshka Quantization (MatQuant) is a recent quantization approach showing that a single integer-quantized model can be served across multiple precisions, by slicing the most significant bits (MSB) at inference time. T…