Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models
Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual generalization or compression to facilitate transfer to a large number of (potentially unseen) languages. However, these models should ideally also be private, linguistically fair, and transparent, by relating their predictions to training data. Can these requirements be simultaneously satisfied? We show that multilingual compression and linguistic fairness are compatible with differential privacy, but that differential privacy is at odds with training data influence sparsity, an objective for transparency. We further present a series of experiments on two common NLP tasks and evaluate multilingual compression and training data influence sparsity under different privacy guarantees, exploring these trade-offs in more detail. Our results suggest that we need to develop ways to jointly optimize for these objectives in order to find practical trade-offs.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Impact of Differential Privacy on Group Disparity Mitigation
The performance cost of differential privacy has, for some applications, been shown to be higher for minority groups; fairness, conversely, has been shown to disproportionally compromise the privacy of members of such gr…
FairnessFair Differentially Private Federated Learning Framework
Federated learning (FL) is a distributed machine learning strategy that enables participants to collaborate and train a shared model without sharing their individual datasets. Privacy and fairness are crucial considerati…
FairnessFederated LearningDifferentially Private and Fair Classification via Calibrated Functional Mechanism
Machine learning is increasingly becoming a powerful tool to make decisions in a wide variety of applications, such as medical diagnosis and autonomous driving. Privacy concerns related to the training data and unfair be…
Autonomous DrivingBIG-bench Machine LearningClassificationFairness+2Fairness Meets Privacy: Integrating Differential Privacy and Demographic Parity in Multi-class Classification
The increasing use of machine learning in sensitive applications demands algorithms that simultaneously preserve data privacy and ensure fairness across potentially sensitive sub-populations. While privacy and fairness h…
Multi-class ClassificationWhere to Intervene? Benchmarking Fairness-Aware Learning on Differentially Private Synthetic Tabular Data
Machine learning models are increasingly deployed in high-stakes domains, raising concerns about both privacy and fairness. Differential Privacy (DP) has become a gold standard for privacy-preserving data analysis, while…