Adversarial Self-Supervised Data-Free Distillation for Text Classification
Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computationally expensive model to a resource-efficient lightweight model. However, most KD algorithms, especially in NLP, rely on the accessibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two-stage data-free distillation method, named Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed for compressing large-scale transformer-based models (e.g., BERT). To avoid text generation in discrete space, we introduce a Plug & Play Embedding Guessing method to craft pseudo embeddings from the teacher's hidden knowledge. Meanwhile, with a self-supervised module to quantify the student's ability, we adapt the difficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGeneral ClassificationKnowledge Distillationtext-classificationText ClassificationText GenerationSimilar Papers 제목 키워드 기반
DAD++: Improved Data-free Test Time Adversarial Defense
With the increasing deployment of deep neural networks in safety-critical applications such as self-driving cars, medical imaging, anomaly detection, etc., adversarial robustness has become a crucial concern in the relia…
Adversarial DefenseAdversarial RobustnessAnomaly DetectionData-free Knowledge Distillation+4Domain-Agnostic Clustering with Self-Distillation
Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmenta…
ClusteringData AugmentationDeep ClusteringKnowledge Distillation+2Robust and Accurate Object Detection via Self-Knowledge Distillation
Object detection has achieved promising performance on clean datasets, but how to achieve better tradeoff between the adversarial robustness and clean precision is still under-explored. Adversarial training is the mainst…
Adversarial RobustnessKnowledge Distillationobject-detectionObject Detection+2Leave No One Behind: Online Self-Supervised Self-Distillation for Sequential Recommendation
Sequential recommendation methods play a pivotal role in modern recommendation systems. A key challenge lies in accurately modeling user preferences in the face of data sparsity. To tackle this challenge, recent methods …
ClusteringContrastive LearningOnline ClusteringRecommendation Systems+2MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
We present a new method of self-supervised learning and knowledge distillation based on the multi-views and multi-representations (MV-MR). The MV-MR is based on the maximization of dependence between learnable embeddings…
ClusteringContrastive LearningKnowledge DistillationLinear evaluation+3