How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding

2021-09-13 · Tianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma, Ali Ghodsi, Mehdi Rezagholizadeh

Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing (NLP) applications, little is understood about how one KD algorithm compares to another and whether these approaches can be complimentary to each other. In this work, we evaluate various KD algorithms on in-domain, out-of-domain and adversarial testing. We propose a framework to assess the adversarial robustness of multiple KD algorithms. Moreover, we introduce a new KD algorithm, Combined-KD, which takes advantage of two promising approaches (better training scheme and more efficient data augmentation). Our extensive experimental results show that Combined-KD achieves state-of-the-art results on the GLUE benchmark, out-of-domain generalization, and adversarial robustness compared to competitive methods.

📄 PDF Abstract BibTeX arXiv:2109.05696

Code (1)

huawei-noah/kd-nlp pytorch

Tasks

Adversarial RobustnessAllData Augmentation Domain Generalization Knowledge Distillation Model Compression Natural Language Understanding

How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding

2021-11-01 · Findings (EMNLP) 2021 11 · Tianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma 외

Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing…

Adversarial Robustness All Data Augmentation Domain Generalization+3

Ranking and Selection from Pairwise Comparisons: Empirical Bayes Methods for Citation Analysis

2021-12-21 · Jiaying Gu, Roger Koenker

We study the Stigler model of citation flows among journals adapting the pairwise comparison model of Bradley and Terry to do ranking and selection of journal influence based on nonparametric empirical Bayes procedures. …

An Empirical Study of Accuracy, Fairness, Explainability, Distributional Robustness, and Adversarial Robustness

2021-09-29 · Moninder Singh, Gevorg Ghalachyan, Kush R. Varshney, Reginald E. Bryant

To ensure trust in AI models, it is becoming increasingly apparent that evaluation of models must be extended beyond traditional performance metrics, like accuracy, to other dimensions, such as fairness, explainability, …

Adversarial Robustness Fairness

An Empirical Analysis of Federated Learning Models Subject to Label-Flipping Adversarial Attack

2024-12-24 · Kunal Bhatnagar, Sagana Chattanathan, Angela Dang, Bhargav Eranki 외

In this paper, we empirically analyze adversarial attacks on selected federated learning models. The specific learning models considered are Multinominal Logistic Regression (MLR), Support Vector Classifier (SVC), Multil…

Adversarial Attack Federated Learning

Ranking XAI Methods for Head and Neck Cancer Outcome Prediction

2026-04-17 · Baoqiang Ma, Djennifer K. Madzia-Madzou, Rosa C. J. Kraaijveld, Jin Ouyang arxiv

For head and neck cancer (HNC) patients, prognostic outcome prediction can support personalized treatment strategy selection. Improving prediction performance of HNC outcomes has been extensively explored by using advanc…