Weak-to-Strong Knowledge Distillation Accelerates Visual Learning
Large-scale visual learning is increasingly limited by training cost. Existing knowledge distillation methods transfer from a stronger teacher to a weaker student for compression or final-accuracy improvement. We instead investigate distillation to accelerate the training of strong students. We propose a generalizable plug-and-play recipe that freezes a weaker teacher, applies distillation only in early training, and turns it off once the student reaches and surpasses teacher-level performance. For ImageNet and CIFAR classification, this strategy reaches target thresholds much earlier, with up to 4.8 times speedup measured by epochs. We confirm that the method generalizes to other tasks and report 1.7 times epoch speedup for object detection on the COCO dataset, and 2.5 times earlier target-FID crossing for diffusion generation on the CIFAR-10 dataset, measured in steps. These findings validate our method as a universal speedup mechanism for visual learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationObject DetectionSimilar Papers 제목 키워드 기반
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, wher…
Strong Teacher Not Needed? On Distillation in LLM Pretraining
Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By va…
Knowledge DistillationSynergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
Brain-inspired spiking neural networks (SNNs) promise to be a low-power alternative to computationally intensive artificial neural networks (ANNs), although performance gaps persist. Recent studies have improved the perf…
Knowledge DistillationAdversarial RobustnessWeak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight
As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We …
Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidance
Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. Meanwhile, current…
Domain AdaptationKnowledge DistillationLanguage ModellingSelf-Knowledge Distillation+1