Can Model Compression Improve NLP Fairness
Model compression techniques are receiving increasing attention; however, the effect of compression on model fairness is still under explored. This is the first paper to examine the effect of distillation and pruning on the toxicity and bias of generative language models. We test Knowledge Distillation and Pruning methods on the GPT2 model and found a consistent pattern of toxicity and bias reduction after model distillation; this result can be potentially interpreted by existing line of research which describes model compression as a regularization technique; our work not only serves as a reference for safe deployment of compressed models, but also extends the discussion of "compression as regularization" into the setting of neural LMs, and hints at the possibility of using compression to develop fairer models.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessKnowledge DistillationmodelModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Privacy, Interpretability, and Fairness in the Multilingual Space
Multilingual generalization or compression is an objective for cross-lingual models in natural language processing (NLP). We explore how the compression sought for in such models aligns with other common objectives in NL…
FairnessRetrievalSentenceSentence RetrievalFairLRF: Achieving Fairness through Sparse Low Rank Factorization
As deep learning (DL) techniques become integral to various applications, ensuring model fairness while maintaining high performance has become increasingly critical, particularly in sensitive fields such as medical diag…
Model CompressionMedical DiagnosisThe Effect of Model Compression on Fairness in Facial Expression Recognition
Deep neural networks have proved hugely successful, achieving human-like performance on a variety of tasks. However, they are also computationally expensive, which has motivated the development of model compression techn…
Facial Expression RecognitionFacial Expression Recognition (FER)FairnessModel CompressionGoing Beyond Classification Accuracy Metrics in Model Compression
With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the…
ClassificationEdge-computingFairnessKnowledge Distillation+2What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias
Knowledge Distillation is a commonly used Deep Neural Network (DNN) compression method, which often maintains overall generalization performance. However, we show that even for balanced image classification datasets, suc…
Age/UnbiasedFairnessGender Bias Detectionimage-classification+6