Auto-tuning Neural Network Quantization Framework for Collaborative Inference Between the Cloud and Edge
Recently, deep neural networks (DNNs) have been widely applied in mobile intelligent applications. The inference for the DNNs is usually performed in the cloud. However, it leads to a large overhead of transmitting data via wireless network. In this paper, we demonstrate the advantages of the cloud-edge collaborative inference with quantization. By analyzing the characteristics of layers in DNNs, an auto-tuning neural network quantization framework for collaborative inference is proposed. We study the effectiveness of mixed-precision collaborative inference of state-of-the-art DNNs by using ImageNet dataset. The experimental results show that our framework can generate reasonable network partitions and reduce the storage on mobile devices with trivial loss of accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Collaborative InferenceQuantizationSimilar Papers 제목 키워드 기반
Learning to Quantize Deep Neural Networks: A Competitive-Collaborative Approach
Reducing the model size and computation costs for dedicated AI accelerator designs, neural network quantization methods have at- tracted momentous attention recently. Unfortunately, merely minimizing quantization loss…
QuantizationAutomated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline
Multiple Deep Neural Networks (DNNs) integrated into single Deep Learning (DL) inference pipelines e.g. Multi-Task Learning (MTL) or Ensemble Learning (EL), etc., albeit very accurate, pose challenges for edge deployment…
Ensemble LearningMulti-Task LearningQuantizationEfficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
With the breakthrough progress of large language models (LLMs) in natural language processing and multimodal tasks, efficiently deploying them on resource-constrained edge devices has become a critical challenge. The Mix…
Quantization-Aware Collaborative Inference for Large Embodied AI Models
Large artificial intelligence models (LAIMs) are increasingly regarded as a core intelligence engine for embodied AI applications. However, the massive parameter scale and computational demands of LAIMs pose significant …
Generative Design of Hardware-aware DNNs
To efficiently run DNNs on the edge/cloud, many new DNN inference accelerators are being designed and deployed frequently. To enhance the resource efficiency of DNNs, model quantization is a widely-used approach. However…
Quantization