Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer

2025-09-25 · Abdur Rehman, S M A Sharif, Md Abdur Rahaman, Mohamed Jismy Aashik Rasool, Seongwan Kim, Jaeho Lee arxiv

Quantization-aware training (QAT) combined with knowledge distillation (KD) is a promising strategy for compressing Artificial Intelligence (AI) models for deployment on resource-constrained hardware. However, existing QAT-KD methods often struggle to balance task-specific (TS) and distillation losses due to heterogeneous gradient magnitudes, especially under low-bit quantization. We propose Game of Regularizer (GoR), a novel learnable regularization method that adaptively balances TS and KD objectives using only two trainable parameters for dynamic loss weighting. GoR reduces conflict between supervision signals, improves convergence, and boosts the performance of small quantized models (SQMs). Experiments on image classification, object detection (OD), and large language model (LLM) compression show that GoR consistently outperforms state-of-the-art QAT-KD methods. On low-power edge devices, it delivers faster inference while maintaining full-precision accuracy. We also introduce QAT-EKD-GoR, an ensemble distillation framework that uses multiple heterogeneous teacher models. Under optimal conditions, the proposed EKD-GoR can outperform full-precision models, providing a robust solution for real-world deployment.

📄 PDF Abstract BibTeX arXiv:2509.20854

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Image Classification Object Detection

Quantization Robustness of Monotone Operator Equilibrium Networks

2026-03-11 · James Li, Philip H. W. Leong, Thomas Chaffey arxiv

Monotone operator equilibrium networks are implicit-layer models whose output is the unique equilibrium of a monotone operator, guaranteeing existence, uniqueness, and convergence. When deployed on low-precision hardware…

Model compression via distillation and quantization

2018-02-15 · ICLR 2018 1 · Antonio Polino, Razvan Pascanu, Dan Alistarh

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently…

image-classification model Model Compression Quantization+1

Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

2020-09-30 · Yoonho Boo, Sungho Shin, Jungwook Choi, Wonyong Sung

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In…

image-classification Image Classification Knowledge Distillation Quantization+3

Quantized Feature Distillation for Network Quantization

2023-07-20 · Ke Zhu, Yin-Yin He, Jianxin Wu

Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid gro…

image-classification Image Classification Image Segmentation Knowledge Distillation+5

Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery

2026-01-27 · Meng Xin, Sweta Priyadarshi, Jingyu Xin, Bilal Kartal 외 arxiv

This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-p…

Reinforcement Learning