Papers Data-free Knowledge Distillation
“Data-free Knowledge Distillation” 태그가 달린 논문 89편 · 필터 해제
Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation
The deployment of Vision-Language Models (VLMs) in critical domains like disaster management requires high-quality multimodal datasets, especially for transferring knowledge via Data-Free Knowledge Distillation (DFKD). H…
Data-free Knowledge DistillationOn the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) ac…
Data-free Knowledge DistillationGradient Transformer: Learning to Generate Updates for LLMs
Many organizations lack computational resources to fine-tune large language models (LLMs) on private (unshareable) data for better utility, while fine-tuning tiny language models (TinyLMs) alone performs poorly. To addre…
Data-free Knowledge DistillationSTARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
SNNs promise energy-efficient and low-latency inference, but their performance still trails that of ANNs. ANN-to-SNN knowledge distillation helps narrow this gap, yet the original training data are often unavailable in p…
Data-free Knowledge DistillationDiverse Image Priors for Black-box Data-free Knowledge Distillation
Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprie…
Data-free Knowledge DistillationContrastive LearningTabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins
Data-free knowledge distillation enables model compression without original training data, critical for privacy-sensitive tabular domains. However, existing methods does not perform well on tabular data because they do n…
Data-free Knowledge DistillationModel CompressionModel extractionFedBiCross: Personalized One-Shot Federated Learning on Medical Images
Data-free knowledge distillation-based one-shot federated learning (OSFL) trains a model in a single communication round without sharing raw data, making OSFL attractive for privacy-sensitive medical applications. Howeve…
Data-free Knowledge DistillationFederated LearningSeeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation
Semantic segmentation requires a holistic understanding of the physical world, as it assigns semantic labels to spatially continuous and structurally coherent objects rather than to isolated pixels. However, existing dat…
Data-free Knowledge DistillationSemantic SegmentationPost-Pruning Accuracy Recovery via Data-Free Knowledge Distillation
Model pruning is a widely adopted technique to reduce the computational complexity and memory footprint of Deep Neural Networks (DNNs). However, global unstructured pruning often leads to significant degradation in accur…
Data-free Knowledge DistillationModel CompressionReviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its scalability is limited by synchronization overhead. Asynchronous federated learning (AFL) alleviat…
Data-free Knowledge DistillationFederated LearningConditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
Data-free knowledge distillation~(DFKD) is an effective manner to solve model compression and transmission restrictions while retaining privacy protection, which has attracted extensive attention in recent years. Current…
Data-free Knowledge DistillationContrastive LearningModel CompressionPRISM: Precision-Recall Informed Data-Free Knowledge Distillation via Generative Diffusion
Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access to the real in-distribution (ID) data. While existing methods perform well on small-scale images, they suffer from mo…
Data-free Knowledge DistillationDomain GeneralizationPrompt EngineeringC2G-KD: PCA-Constrained Generator for Data-Free Knowledge Distillation
We introduce C2G-KD, a data-free knowledge distillation framework where a class-conditional generator is trained to produce synthetic samples guided by a frozen teacher model and geometric constraints derived from PCA. T…
Data-free Knowledge DistillationWhen Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access the real in-distribution (ID) data. Its common solution is to use a generator to synthesize fake data and use them as…
Data-free Knowledge DistillationAdversarial RobustnessMosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
Federated Learning (FL) is a decentralized machine learning paradigm that enables clients to collaboratively train models while preserving data privacy. However, the coexistence of model and data heterogeneity gives rise…
Data-free Knowledge DistillationFederated Learningimage-classificationImage Classification+3CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation
Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primaril…
Data-free Knowledge DistillationKnowledge DistillationTransfer LearningAdversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks
Data-free Knowledge Distillation (DFKD) is a method that constructs pseudo-samples using a generator without real data, and transfers knowledge from a teacher model to a student by enforcing the student to overcome dimen…
Data-free Knowledge DistillationKnowledge DistillationHFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast
Most current federated learning frameworks are modeled as static processes, ignoring the dynamic characteristics of the learning system. Under the limited communication budget of the central server, the flexible model ar…
Data-free Knowledge DistillationFederated LearningKnowledge DistillationTransfer LearningDoes Training with Synthetic Data Truly Protect Privacy?
As synthetic data becomes increasingly popular in machine learning tasks, numerous methods--without formal differential privacy guarantees--use synthetic data for training. These methods often claim, either explicitly or…
Data-free Knowledge DistillationDataset DistillationKnowledge DistillationHybrid Data-Free Knowledge Distillation
Data-free knowledge distillation aims to learn a compact student network from a pre-trained large teacher network without using the original training data of the teacher network. Existing collection-based and generation-…
Data-free Knowledge DistillationGenerative Adversarial NetworkKnowledge DistillationRepresentation Learning