Online Ensemble Model Compression using Knowledge Distillation
This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models. Each model learns unique representations from the data distribution due to its distinct architecture. This helps the ensemble generalize better by combining every model's knowledge. The distilled students and ensemble teacher are trained simultaneously without requiring any pretrained weights. Moreover, our proposed method can deliver multi-compressed students with single training, which is efficient and flexible for different scenarios. We provide comprehensive experiments using state-of-the-art classification models to validate our framework's effectiveness. Notably, using our framework a 97% compressed ResNet110 student model managed to produce a 10.64% relative accuracy gain over its individual baseline training on CIFAR100 dataset. Similarly a 95% compressed DenseNet-BC(k=12) model managed a 8.17% relative accuracy gain.
Code (1)
Tasks
Knowledge DistillationmodelModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression
Knowledge distillation (KD) is an effective model compression technique where a compact student network is taught to mimic the behavior of a complex and highly trained teacher network. In contrast, Mutual Learning (ML) p…
Knowledge DistillationModel Compressionobject-detectionObject Detection+1Distilling Model Knowledge
Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Id…
Bayesian InferenceBIG-bench Machine LearningKnowledge Distillationmodel+1Semi-Supervised Learning with Online Knowledge Distillation for Skin Lesion Classification
Deep Learning has emerged as a promising approach for skin lesion analysis. However, existing methods mostly rely on fully supervised learning, requiring extensive labeled data, which is challenging and costly to obtain.…
Skin Lesion ClassificationKnowledge DistillationEnsemble LearningMixed Sample Augmentation for Online Distillation
Mixed Sample Regularization (MSR), such as MixUp or CutMix, is a powerful data augmentation strategy to generalize convolutional neural networks. Previous empirical analysis has illustrated an orthogonal performance gain…
Data AugmentationKnowledge DistillationPILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely…
Knowledge Distillation