SDQ: Sparse Decomposed Quantization for LLM Inference

2024-06-19 · Geonhwa Jeong, Po-An Tsai, Stephen W. Keckler, Tushar Krishna

Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions to trillions of parameters, which hinder the wide adaptation of those models due to their extremely large compute and memory requirements. To resolve the issue, various model compression methods are being actively investigated. In this work, we propose SDQ (Sparse Decomposed Quantization) to exploit both structured sparsity and quantization to achieve both high compute and memory efficiency. From our evaluations, we observe that SDQ can achieve 4x effective compute throughput with <1% quality drop.

📄 PDF Abstract BibTeX arXiv:2406.13868

Code (0)

등록된 구현이 없습니다.

Tasks

Model Compression Quantization

Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

2024-06-16 · Jungi Lee, Wonbeom Lee, Jaewoong Sim

Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning and have thus become one of the most important workloads in today's computing landscape. However, deploying LLM infere…

Quantization Tensor Decomposition

Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation

2026-01-30 · Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei 외 arxiv

Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demonstrate that element-wise quantization can…

Composite Code Sparse Autoencoders for first stage retrieval

2022-04-14 · Carlos Lassance, Thibault Formal, Stephane Clinchant

We propose a Composite Code Sparse Autoencoder (CCSA) approach for Approximate Nearest Neighbor (ANN) search of document representations based on Siamese-BERT models. In Information Retrieval (IR), the ranking pipeline i…

Image Retrieval Information Retrieval Quantization Retrieval

Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization

2023-09-07 · Habib Hajimolahoseini, Walid Ahmed, Yang Liu

Low Rank Decomposition (LRD) is a model compression technique applied to the weight tensors of deep learning models in order to reduce the number of trainable parameters and computational complexity. However, due to high…