Run-Time Efficient RNN Compression for Inference on Edge Devices
Recurrent neural networks can be large and compute-intensive, yet many applications that benefit from RNNs run on small devices with very limited compute and storage capabilities while still having run-time constraints. As a result, there is a need for compression techniques that can achieve significant compression without negatively impacting inference run-time and task accuracy. This paper explores a new compressed RNN cell implementation called Hybrid Matrix Decomposition (HMD) that achieves this dual objective. This scheme divides the weight matrix into two parts - an unconstrained upper half and a lower half composed of rank-1 blocks. This results in output features where the upper sub-vector has "richer" features while the lower-sub vector has "constrained features". HMD can compress RNNs by a factor of 2-4x while having a faster run-time than pruning (Zhu &Gupta, 2017) and retaining more model accuracy than matrix factorization (Grachev et al., 2017). We evaluate this technique on 5 benchmarks spanning 3 different applications, illustrating its generality in the domain of edge computing.
Code (0)
등록된 구현이 없습니다.
Tasks
Edge-computingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
Edge devices are increasingly utilized for deploying deep learning applications on embedded systems. The real-time nature of many applications and the limited resources of edge devices necessitate latency-targeted neural…
Neural Network CompressionModel CompressionHardware-Aware DNN Compression for Homogeneous Edge Devices
Deploying deep neural networks (DNNs) across homogeneous edge devices (the devices with the same SKU labeled by the manufacturer) often assumes identical performance among them. However, once a device model is widely dep…
Dynamic Compression Ratio Selection for Edge Inference Systems with Hard Deadlines
Implementing machine learning algorithms on Internet of things (IoT) devices has become essential for emerging applications, such as autonomous driving, environment monitoring. But the limitations of computation capabili…
Autonomous DrivingBIG-bench Machine LearningData CompressionFailure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices
The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constraine…
Knowledge DistillationModel CompressionDeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge
Vision Transformers (ViTs) excel in tackling complex vision tasks, yet their substantial size poses significant challenges for applications on resource-constrained edge devices. The increased size of these models lea…
Model Compression