Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models

2023-07-12 · James O' Neill, Sourav Dutta

We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimizes accumulative quantization errors and outperforms baselines. We apply SDQ to multilingual models XLM-R-Base and InfoXLM-Base and demonstrate that both models can be reduced from 32-bit floating point weights to 8-bit integer weights while maintaining a high level of performance on the XGLUE benchmark. Our results also highlight the challenges of quantizing multilingual models, which must generalize to languages they were not fine-tuned on.

📄 PDF Abstract BibTeX arXiv:2307.05972

Code (0)

등록된 구현이 없습니다.

Tasks

QuantizationXLM-R

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Position-Wise Feed-Forward Layer 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Distilled Low Rank Neural Radiance Field with Quantization for Light Field Compression

2022-07-30 · Jinglei Shi, Christine Guillemot

We propose in this paper a Quantized Distilled Low-Rank Neural Radiance Field (QDLR-NeRF) representation for the task of light field compression. While existing compression methods encode the set of light field sub-apert…

NeRF Quantization

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models

2022-10-08 · Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo 외

There are growing interests in adapting large-scale language models using parameter-efficient fine-tuning methods. However, accelerating the model itself and achieving better inference efficiency through model compressio…

Language Modeling Language Modelling Model Compression parameter-efficient fine-tuning+1

The Hidden Costs of Translation Accuracy: Distillation, Quantization, and Environmental Impact

2025-09-28 · Dhaathri Vijay, Anandaswarup Vadapalli arxiv

The rapid expansion of large language models (LLMs) has heightened concerns about their computational and environmental costs. This study investigates the trade-offs between translation quality and efficiency by comparin…

Machine Translation Model Compression

LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

2022-06-20 · Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee 외

Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models int…

GPU Quantization Self-Supervised Learning

Layer-Wise Data-Free CNN Compression

2020-11-18 · Maxwell Horton, Yanzi Jin, Ali Farhadi, Mohammad Rastegari

We present a computationally efficient method for compressing a trained neural network without using real data. We break the problem of data-free network compression into independent layer-wise compressions. We show how …

Quantization