Benchmarking Differential Privacy and Federated Learning for BERT Models
Natural Language Processing (NLP) techniques can be applied to help with the diagnosis of medical conditions such as depression, using a collection of a person's utterances. Depression is a serious medical illness that can have adverse effects on how one feels, thinks, and acts, which can lead to emotional and physical problems. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training models with such data. In this work, we study the effects that the application of Differential Privacy (DP) has, in both a centralized and a Federated Learning (FL) setup, on training contextualized language models (BERT, ALBERT, RoBERTa and DistilBERT). We offer insights on how to privately train NLP models and what architectures and setups provide more desirable privacy utility trade-offs. We envisage this work to be used in future healthcare and mental health studies to keep medical history private. Therefore, we provide an open-source implementation of this work.
Code (1)
Tasks
BenchmarkingFederated LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning
Privacy is important considering the financial Domain as such data is highly confidential and sensitive. Natural Language Processing (NLP) techniques can be applied for text classification and entity detection purposes i…
Federated LearningSentiment Analysistext-classificationText ClassificationDifferential Privacy for Adaptive Weight Aggregation in Federated Tumor Segmentation
Federated Learning (FL) is a distributed machine learning approach that safeguards privacy by creating an impartial global model while respecting the privacy of individual client data. However, the conventional FL method…
BenchmarkingBrain Tumor SegmentationFederated LearningImage Segmentation+4Differentially Private Federated Learning: A Systematic Review
In recent years, privacy and security concerns in machine learning have promoted trusted federated learning to the forefront of research. Differential privacy has emerged as the de facto standard for privacy protection i…
Federated LearningPrivacy PreservingAnalysis of Privacy Leakage in Federated Large Language Models
With the rapid adoption of Federated Learning (FL) as the training and tuning protocol for applications utilizing Large Language Models (LLMs), recent research highlights the need for significant modifications to FL to a…
Federated LearningFederated $f$-Differential Privacy
Federated learning (FL) is a training paradigm where the clients collaboratively learn models by repeatedly sharing information without compromising much on the privacy of their local sensitive data. In this paper, we in…
Federated Learning