Papers Knowledge Distillation
“Knowledge Distillation” 태그가 달린 논문 5,291편 · 필터 해제
CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations
Weight-space networks operate directly on parameters of other neural networks, enabling tasks such as predicting model properties, editing trained models, and generating weights. Weight-space symmetries such as neuron pe…
Knowledge DistillationImage ClassificationModel CompressionDistillation of Tabular Foundation Models into Efficient Predictors
Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by tr…
Knowledge DistillationSmaller Models, Better Rejects: Preference Distillation Scaling
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come…
Knowledge DistillationMathematical ReasoningCode GenerationWhat Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide Imaging
Cross-resolution knowledge distillation aims to improve low-magnification whole- slide analysis by transferring high-magnification representations, yet the conditions for useful transfer remain unclear. We develop a deco…
Knowledge DistillationDo We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation …
Knowledge DistillationPersonalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head A…
Personalized Federated LearningKnowledge DistillationOn-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data
Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in current vision-language distillation methods is typically constructed from …
Knowledge DistillationDeep Neural Networks for Learning Intent from sEMG Signals to Support Hardware Devices for Post-Stroke Neurorehabilitation
Finger-specific motor intent is a clinically meaningful control signal for post-stroke neurorehabilitation, where residual muscle activity may remain measurable despite weak or incomplete movement. We study five-finger m…
Knowledge DistillationPretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification
Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicting conclusions about CNNs versus transformers. We present a controll…
Knowledge DistillationPersistent Teacher Anchoring for Tool-Using Agents
Distillation is common in LLM post-training, where on-policy knowledge distillation (OPKD) uses student-generated trajectories to prepare the student for downstream RL. At each state, the student matches a next-token dis…
Knowledge DistillationImportance-Aware Low-Rank Distillation of Diffusion Transformers
Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SV…
Text-to-Image GenerationKnowledge DistillationKnowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across training stages remains unclear. Through contr…
Self-Supervised LearningKnowledge DistillationDual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models
Source-Fully-Free Domain Adaptation (SFF-DA) has emerged as a strategic paradigm to adapt Vision-Language Models (VLMs) without any access to source data or task-specific source models. However, we identify a critical Du…
Knowledge DistillationDomain AdaptationRelational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently …
Knowledge DistillationSpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models
Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A pract…
Knowledge DistillationKnowledge Distillation Driven Semantic NOMA with GAN Refinement for 6G Robotic Vehicle Networks
To achieve sustainable intelligent mobility, 6G-empowered robotic vehicles (RVs) require high-fidelity visual perception under stringent bandwidth and energy constraints. Semantic communication offers a spectral-efficien…
Knowledge DistillationSemantic CommunicationMitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion
Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isol…
Knowledge DistillationCross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification
Automated tea leaf disease classification supports precision agriculture, yet deploying accurate models on edge devices remains challenging under tight compute budgets. Self-supervised vision foundation models such as DI…
Knowledge DistillationCross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data
Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has …
Data-to-Text GenerationKnowledge DistillationKnowledge GraphsSelFusion: Self-distillation for Diffusion Language Models
Diffusion language models (DLMs) alleviate the inherent latency bottleneck of autoregressive (AR) large language models (LLMs), but their degraded generation quality limits practical applicability. Although knowledge dis…
Knowledge Distillation