Microdosing: Knowledge Distillation for GAN based Compression
Recently, significant progress has been made in learned image and video compression. In particular the usage of Generative Adversarial Networks has lead to impressive results in the low bit rate regime. However, the model size remains an important issue in current state-of-the-art proposals and existing solutions require significant computation effort on the decoding side. This limits their usage in realistic scenarios and the extension to video compression. In this paper, we demonstrate how to leverage knowledge distillation to obtain equally capable image decoders at a fraction of the original number of parameters. We investigate several aspects of our solution including sequence specialization with side information for image coding. Finally, we also show how to transfer the obtained benefits into the setting of video compression. Overall, this allows us to reduce the model size by a factor of 20 and to achieve 50% reduction in decoding time.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationVideo CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Functional Perspective on Knowledge Distillation in Neural Networks
Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge dis…
Knowledge DistillationA Unified Knowledge Distillation Framework for Deep Directed Graphical Models
Knowledge distillation (KD) is a technique that transfers the knowledge from a large teacher network to a small student network. It has been widely applied to many different tasks, such as model compression and federate…
Continual LearningFederated LearningKnowledge DistillationModel CompressionWeight Distillation: Transferring the Knowledge in Neural Network Parameters
Knowledge distillation has been proven to be effective in model acceleration and compression. It allows a small network to learn to generalize in the same way as a large network. Recent successes in pre-training suggest …
Knowledge DistillationMachine TranslationTranslationSpirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer
Recent applications pose requirements of both cross-domain knowledge transfer and model compression to machine learning models due to insufficient training data and limited computational resources. In this paper, we prop…
General KnowledgeKnowledge DistillationModel CompressionTransfer LearningModel compression using knowledge distillation with integrated gradients
Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with integrated gradients (IG) as a data augmentation strateg…
Data AugmentationKnowledge DistillationModel Compression