AntMan: Sparse Low-Rank Compression to Accelerate RNN inference
Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements. To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy. AntMan extends knowledge distillation based training to learn the compressed models efficiently. Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models. Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-of-art. Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationLow-rank compressionMachine Reading ComprehensionReading ComprehensionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footp…
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to…
QuantizationEndor: Hardware-Friendly Sparse Format for Offloaded LLM Inference
The increasing size of large language models (LLMs) challenges their usage on resource-constrained platforms. For example, memory on modern GPUs is insufficient to hold LLMs that are hundreds of Gigabytes in size. Offloa…
CPUGPUOn the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
Visual token compression is widely used to accelerate large vision-language models (LVLMs) by pruning or merging visual tokens, yet its adversarial robustness remains unexplored. We show that existing encoder-based attac…
Adversarial RobustnessConvolutional neural networks compression with low rank and sparse tensor decompositions
Convolutional neural networks show outstanding results in a variety of computer vision tasks. However, a neural network architecture design usually faces a trade-off between model performance and computational/memory com…
CPUimage-classificationImage ClassificationNeural Network Compression