Representation Consolidation for Training Expert Students
Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher. A more useful goal than emulation, yet under-explored, is for the student to learn feature representations that transfer well to future tasks. However, we observe that standard distillation of task-specific teachers actually *reduces* the transferability of student representations to downstream tasks. We show that a multi-head, multi-task distillation method using an unlabeled proxy dataset and a generalist teacher is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance, outperforming the teacher(s) and the strong baseline of ImageNet pretrained features. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model, whose representation is improved on all teachers' domain(s).
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
School choice with independent versus consolidated districts
This paper studies the welfare effects of school district consolidation. Using incomplete rank-ordered lists (ROLs) submitted for admission to the Hungarian secondary school system, we estimate complete ROLs assuming tha…
counterfactualBatch Model Consolidation: A Multi-Task Model Consolidation Framework
In Continual Learning (CL), a model is required to learn a stream of tasks sequentially without significant performance degradation on previously learned tasks. Current approaches fail for a long sequence of tasks from d…
Continual Learningimage-classificationImage ClassificationmodelPersistent Memory Through Triple-Loop Consolidation in a Non-Gradient Dissipative Cognitive Architecture
Dissipative cognitive architectures maintain computation through continuous energy expenditure, where units that exhaust their energy are stochastically replaced with fresh random state. This creates a fundamental challe…
Fewer, better pathways for all? Intersectional impacts of rural school consolidation in China's minority regions
Primary school consolidation--the closure of small community schools or their mergers into larger, better-resourced schools--is emerging as a significant policy response to changing demographics in middle income countrie…
AllConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression
Mixture-of-Experts (MoE) language models reduce per-token computation but still require storing and serving all experts, making deployment memory-intensive. Existing post-training compression methods mainly shrink this c…