Towards Fully 8-bit Integer Inference for the Transformer Model

2020-09-17 · Ye Lin, Yanyang Li, Tengbo Liu, Tong Xiao, Tongran Liu, Jingbo Zhu

8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e.g., Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En<->Ro, WMT14 En<->De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.

📄 PDF Abstract BibTeX arXiv:2009.08034

Code (0)

등록된 구현이 없습니다.

Tasks

Language Modelling Quantization Translation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Faster Inference of Integer SWIN Transformer by Removing the GELU Activation

2024-02-02 · Mohammadreza Tayaranian, Seyyed Hasan Mozafari, James J. Clark, Brett Meyer 외

SWIN transformer is a prominent vision transformer model that has state-of-the-art accuracy in image classification tasks. Despite this success, its unique architecture causes slower inference compared with similar deep …

GPU image-classification Image Classification Knowledge Distillation+1

IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers

2025-11-19 · Gihwan Kim, Jemin Lee, Hyungshin Kim arxiv

Previous Quantization-Aware Training (QAT) methods for vision transformers rely on expensive retraining to recover accuracy loss in non-linear layer quantization, limiting their use in resource-constrained environments. …

Image Classification Object Detection

I-BERT: Integer-only BERT Quantization

2021-01-05 · Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney 외

Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive…

GPU Natural Language Inference Natural Language Understanding Quantization

I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation

2025-09-12 · Jordan Sassoon, Michal Szczepanski, Martyna Poreba arxiv

Vision Transformers (ViTs) have recently achieved strong results in semantic segmentation, yet their deployment on resource-constrained devices remains limited due to their high memory footprint and computational cost. Q…

Semantic Segmentation

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

2022-07-04 · ICCV 2023 1 · Zhikai Li, Qingyi Gu

Vision Transformers (ViTs) have achieved state-of-the-art performance on various computer vision applications. However, these models have considerable storage and computational overheads, making their deployment and effi…

Quantization