Cramnet: Layer-wise Deep Neural Network Compression with Knowledge Transfer from a Teacher Network
Neural Networks accomplish amazing things, but they suffer from computational and memory bottlenecks that restrict their usage. Nowhere can this be better seen than in the mobile space, where specialized hardware is being created just to satisfy the demand for neural networks. Previous studies have shown that neural networks have vastly more connections than they actually need to do their work. This thesis develops a method that can compress networks to less than 10% of memory and less than 25% of computational power, without loss of accuracy, and without creating sparse networks that require special code to run.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Network CompressionTransfer LearningSimilar Papers 제목 키워드 기반
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
In this work, we propose a novel approach for layerwise representation learning of a trained neural network. In particular, we form a Bregman divergence based on the layer's transfer function and construct an extension o…
Knowledge DistillationRepresentation LearningMICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression
State-of-the-art deep model compression methods exploit the low-rank approximation and sparsity pruning to remove redundant parameters from a learned hidden layer. However, they process each hidden layer individually whi…
Knowledge DistillationModel CompressionMulti-Granularity Structural Knowledge Distillation for Language Model Compression
Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity language units (e.g., token-level or sam…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression+1DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provid…
Knowledge DistillationGreedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
Fine-tuning transformer models after unsupervised pre-training reaches a very high performance on many different natural language processing tasks. Unfortunately, transformers suffer from long inference times which great…
Knowledge DistillationUnsupervised Pre-training