Knowledge Distillation
7개 벤치마크 · 논문 5,291편 · 이 태스크의 논문 보기 →
Benchmarks
Most implemented
Focal Loss for Dense Object Detection
Distilling the Knowledge in a Neural Network
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
Papers
CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations
Weight-space networks operate directly on parameters of other neural networks, enabling tasks such as predicting model properties, editing trained models, and generating weights. Weight-space symmetries such as neuron pe…
Knowledge DistillationImage ClassificationModel CompressionDistillation of Tabular Foundation Models into Efficient Predictors
Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by tr…
Knowledge DistillationSmaller Models, Better Rejects: Preference Distillation Scaling
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come…
Knowledge DistillationMathematical ReasoningCode GenerationWhat Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide Imaging
Cross-resolution knowledge distillation aims to improve low-magnification whole- slide analysis by transferring high-magnification representations, yet the conditions for useful transfer remain unclear. We develop a deco…
Knowledge DistillationDo We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation …
Knowledge DistillationPersonalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head A…
Personalized Federated LearningKnowledge Distillation