Knowledge Distillation with Noisy Labels for Natural Language Understanding
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of research is the impact of noisy (corrupted) labels on KD. We present, to the best of our knowledge, the first study on KD with noisy labels in Natural Language Understanding (NLU). We document the scope of the problem and present two methods to mitigate the impact of label noise. Experiments on the GLUE benchmark show that our methods are effective even under high noise levels. Nevertheless, our results indicate that more research is necessary to cope with label noise under the KD.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationNatural Language UnderstandingSimilar Papers 제목 키워드 기반
FlyKD: Graph Knowledge Distillation on the Fly with Curriculum Learning
Knowledge Distillation (KD) aims to transfer a more capable teacher model's knowledge to a lighter student model in order to improve the efficiency of the model, making it faster and more deployable. However, the student…
Knowledge DistillationBlind Knowledge Distillation for Robust Image Classification
Optimizing neural networks with noisy labels is a challenging task, especially if the label set contains real-world noise. Networks tend to generalize to reasonable patterns in the early training stages and overfit to sp…
Classificationimage-classificationImage ClassificationKnowledge Distillation+1Federated Learning with Extremely Noisy Clients via Negative Distillation
Federated learning (FL) has shown remarkable success in cooperatively training deep models, while typically struggling with noisy labels. Advanced works propose to tackle label noise by a re-weighting strategy with a str…
Federated LearningKnowledge DistillationStudent as an Inherent Denoiser of Noisy Teacher
Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by t…
Knowledge DistillationLanguage ModelingLanguage ModellingLarge Language Model+1Learning from Noisy Labels with Distillation
The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Traditionally, the label noises have been treated as st…