CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework
Numerous self-supervised learning paradigms, such as contrastive learning and masked image modeling, learn powerful representations from unlabeled data but are typically pretrained in isolation, overlooking complementary insights and yielding large models that are impractical for resource-constrained deployment. To overcome these challenges, we introduce Consensus-oriented Masked Distillation (CoMAD), a lightweight, parameter-free framework that unifies knowledge from multiple current state-of-the-art self-supervised Vision Transformers into a compact student network. CoMAD distills from three pretrained ViT-Base teachers, MAE, MoCo v3, and iBOT, each offering distinct semantic and contextual priors. Rather than naively averaging teacher outputs, we apply asymmetric masking: the student sees only 25 percent of patches while each teacher receives a progressively lighter, unique mask, forcing the student to interpolate missing features under richer contexts. Teacher embeddings are aligned to the student's space via a linear adapter and layer normalization, then fused through our joint consensus gating, which weights each token by combining cosine affinity with inter-teacher agreement. The student is trained with dual-level KL divergence on visible tokens and reconstructed feature maps, capturing both local and global structure. On ImageNet-1K, CoMAD's ViT-Tiny achieves 75.4 percent Top-1, an increment of 0.4 percent over the previous state-of-the-art. In dense-prediction transfers, it attains 47.3 percent mIoU on ADE20K, and 44.5 percent box average precision and 40.5 percent mask average precision on MS-COCO, establishing a new state-of-the-art in compact SSL distillation.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningContrastive LearningResults from the Paper
| Rank | Task | Dataset | Model | Metrics |
|---|---|---|---|---|
| #2 | Contrastive Learning | imagenet-1k | Consensus-oriented | ImageNet Top-1 Accuracy: 75.4 |
Similar Papers 제목 키워드 기반
KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo
Supervised multi-view stereo (MVS) methods have achieved remarkable progress in terms of reconstruction quality, but suffer from the challenge of collecting large-scale ground-truth depth. In this paper, we propose a nov…
Knowledge DistillationSelf-Supervised LearningDMT: Comprehensive Distillation with Multiple Self-supervised Teachers
Numerous self-supervised learning paradigms, such as contrastive learning and masked image modeling, have been proposed to acquire powerful and general representations from unlabeled data. However, these models are commo…
Contrastive LearningModel CompressionSelf-Supervised LearningSimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation
Feature regression is a simple way to distill large neural network models to smaller ones. We show that with simple changes to the network architecture, regression can outperform more complex state-of-the-art approaches …
Knowledge DistillationregressionDistilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment
Recent advances have indicated the strengths of self-supervised pre-training for improving representation learning on downstream tasks. Existing works often utilize self-supervised pre-trained models by fine-tuning on do…
Knowledge DistillationRepresentation LearningSelf-Supervised LearningSelf-supervised Models are Good Teaching Assistants for Vision Transformers
Transformers have shown remarkable progress on computer vision tasks in the past year. Compared to their CNN counterparts, transformers usually need the help of distillation to achieve comparable results on middle or sma…
Image ClassificationKnowledge Distillation