Differentially Private Model Compression
Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost of these models -- which consist of hundreds of millions of parameters -- however, can be prohibitively large. Hence, often in practice, LLMs are compressed before they are deployed in specific applications. In this paper, we initiate the study of differentially private model compression and propose frameworks for achieving 50% sparsity levels while maintaining nearly full performance. We demonstrate these ideas on standard GLUE benchmarks using BERT models, setting benchmarks for future research on this topic.
Code (0)
등록된 구현이 없습니다.
Tasks
modelModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Scalable Differentially Private Data Compression via Diffusion and Stochastic Codes
The ever-increasing collection of personal data has created mounting pressure to develop technologies that protect sensitive aspects of individual identity. Differential privacy (DP) provides a principled framework with …
Image ClassificationGradient Sparsification Can Improve Performance of Differentially-Private Convex Machine Learning
We use gradient sparsification to reduce the adverse effect of differential privacy noise on performance of private machine learning models. To this aim, we employ compressed sensing and additive Laplace noise to evaluat…
BIG-bench Machine Learningcompressed sensingPrivacy PreservingDifferentially Private Knowledge Distillation via Synthetic Text Generation
Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks. However, the increasing urgency of data privacy puts pressure on practitioners to train LLMs with Differential P…
Knowledge DistillationModel CompressionPrivacy PreservingText GenerationOn the Interaction Between Differential Privacy and Gradient Compression in Deep Learning
While differential privacy and gradient compression are separately well-researched topics in machine learning, the study of interaction between these two topics is still relatively new. We perform a detailed empirical st…
SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compression
To enable large-scale machine learning in bandwidth-hungry environments such as wireless networks, significant progress has been made recently in designing communication-efficient federated learning algorithms with the a…
Federated LearningPrivacy Preserving