Post-Training Statistical Calibration for Higher Activation Sparsity

2024-12-10 · Vui Seng Chua, Yujie Pan, Nilesh Jain

We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: https://github.com/IntelLabs/SCAP.

📄 PDF Abstract BibTeX arXiv:2412.07174

Code (1)

intellabs/scap 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Adam 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Position-Wise Feed-Forward Layer 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Enhancing Post-training Quantization Calibration through Contrastive Learning

2024-01-01 · CVPR 2024 1 · Yuzhang Shang, Gaowen Liu, Ramana Rao Kompella, Yan Yan

Post-training quantization (PTQ) converts a pre-trained full-precision (FP) model into a quantized model in a training-free manner. Determining suitable quantization parameters such as scaling factors and weight roun…

Contrastive Learning Quantization

Specificity- and Calibration-Aware Breast Ultrasound Segmentation via Entropy-Guided Boundary Supervision

2026-06-21 · Manar Alsaid, Mandip Shrestha, Mohammad Abbas arxiv

Lesion segmentation in breast ultrasound involves two related challenges. In images with lesions, speckle noise, low tissue contrast, and posterior acoustic shadowing cause boundary leakage and incomplete contour delinea…

Lesion Segmentation

Tail-Aware Post-Training Quantization for 3D Geometry Models

2026-02-02 · Sicheng Pan, Chen Tang, Shuzhao Xie, Ke Yang 외 arxiv

The burgeoning complexity and scale of 3D geometry models pose significant challenges for deployment on resource-constrained platforms. While Post-Training Quantization (PTQ) enables efficient inference without retrainin…

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

2026-05-11 · Chayne Thrash, Ali Abbasi, Soheil Kolouri arxiv

Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult …

Common Sense Reasoning

Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization

2025-10-12 · Bowei He, Lihao Yin, Huiling Zhen, Shuqi Liu 외 arxiv

Post-training compression has been a widely employed approach to scale down large language model (LLM) and facilitate efficient inference. In various proposed compression methods, including pruning and quantization, cali…

Code Generation