How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing (NLP) applications, little is understood about how one KD algorithm compares to another and whether these approaches can be complimentary to each other. In this work, we evaluate various KD algorithms on in-domain, out-of-domain and adversarial testing. We propose a framework to assess adversarial robustness of multiple KD algorithms. Moreover, we introduce a new KD algorithm, Combined-KD, which takes advantage of two promising approaches (better training scheme and more efficient data augmentation). Our extensive experimental results show that Combined-KD achieves state-of-the-art results on the GLUE benchmark, out-of-domain generalization, and adversarial robustness compared to competitive methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessAllData AugmentationDomain GeneralizationKnowledge DistillationModel CompressionNatural Language UnderstandingSimilar Papers 제목 키워드 기반
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing…
Adversarial RobustnessAllData AugmentationDomain Generalization+3On the Effectiveness of Minimal Context Selection for Robust Question Answering
Machine learning models for question-answering (QA), where given a question and a passage, the learner must select some span in the passage as an answer, are known to be brittle. By inserting a single nuisance sentence i…
Adversarial RobustnessQuestion AnsweringSentenceRanking and Selection from Pairwise Comparisons: Empirical Bayes Methods for Citation Analysis
We study the Stigler model of citation flows among journals adapting the pairwise comparison model of Bradley and Terry to do ranking and selection of journal influence based on nonparametric empirical Bayes procedures. …
Selective Adversarial Attacks on LLM Benchmarks
Benchmarking outcomes increasingly govern trust, selection, and deployment of LLMs, yet these evaluations remain vulnerable to semantically equivalent adversarial perturbations. Prior work on adversarial robustness in NL…
Adversarial RobustnessGeneral KnowledgeAn Empirical Study of Accuracy, Fairness, Explainability, Distributional Robustness, and Adversarial Robustness
To ensure trust in AI models, it is becoming increasingly apparent that evaluation of models must be extended beyond traditional performance metrics, like accuracy, to other dimensions, such as fairness, explainability, …
Adversarial RobustnessFairness