Self-Supervised Pretraining for Differentially Private Learning
We demonstrate self-supervised pretraining (SSP) is a scalable solution to deep learning with differential privacy (DP) regardless of the size of available public datasets in image classification. When facing the lack of public datasets, we show the features generated by SSP on only one single image enable a private classifier to obtain much better utility than the non-learned handcrafted features under the same privacy budget. When a moderate or large size public dataset is available, the features produced by SSP greatly outperform the features trained with labels on various complex private datasets under the same private budget. We also compared multiple DP-enabled training frameworks to train a private classifier on the features generated by SSP. Finally, we report a non-trivial utility 25.3\% of a private ImageNet-1K dataset when $\epsilon=3$. Our source code can be found at \url{https://github.com/UnchartedRLab/SSP}.
Code (1)
Tasks
image-classificationImage ClassificationSimilar Papers 제목 키워드 기반
The role of self-supervised pretraining in differentially private medical image analysis
Differential privacy (DP) provides formal protection for sensitive data but typically incurs substantial losses in diagnostic performance. Model initialization has emerged as a critical factor in mitigating this degradat…
Self-Supervised LearningPosition: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
The performance of differentially private machine learning can be boosted significantly by leveraging the transfer learning capabilities of non-private models pretrained on large public datasets. We critically review thi…
PositionPrivacy PreservingTransfer LearningDP$^2$-VAE: Differentially Private Pre-trained Variational Autoencoders
Modern machine learning systems achieve great success when trained on large datasets. However, these datasets usually contain sensitive information (e.g. medical records, face images), leading to serious privacy concerns…
Large-Scale Differentially Private BERT
In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch size to millions (i.e., mega-batches) i…
Language ModelingLanguage ModellingDifferentially Private Correlation Clustering
Correlation clustering is a widely used technique in unsupervised machine learning. Motivated by applications where individual privacy is a concern, we initiate the study of differentially private correlation clustering.…
BIG-bench Machine LearningClustering