Evaluating Differentially Private Generation of Domain-Specific Text
Generative AI offers transformative potential for high-stakes domains such as healthcare and finance, yet privacy and regulatory barriers hinder the use of real-world data. To address this, differentially private synthetic data generation has emerged as a promising alternative. In this work, we introduce a unified benchmark to systematically evaluate the utility and fidelity of text datasets generated under formal Differential Privacy (DP) guarantees. Our benchmark addresses key challenges in domain-specific benchmarking, including choice of representative data and realistic privacy budgets, accounting for pre-training and a variety of evaluation metrics. We assess state-of-the-art privacy-preserving generation methods across five domain-specific datasets, revealing significant utility and fidelity degradation compared to real data, especially under strict privacy constraints. These findings underscore the limitations of current approaches, outline the need for advanced privacy-preserving data sharing methods and set a precedent regarding their evaluation in realistic scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Synthetic Data GenerationSimilar Papers 제목 키워드 기반
Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains
The difficulty of anonymizing text data hinders the development and deployment of NLP in high-stakes domains that involve private data, such as healthcare and social services. Poorly anonymized sensitive data cannot be e…
FairnessPrivacy PreservingSynthetic Data GenerationLearning and Evaluating a Differentially Private Pre-trained Language Model
Contextual language models have led to significantly better results, especially when pre-trained on the same data as the downstream task. While this additional pre-training usually improves performance, it can lead to in…
Language ModelingLanguage ModellingDP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP…
Synthetic Data GenerationReinforcement LearningComparative Study of Differentially Private Synthetic Data Algorithms from the NIST PSCR Differential Privacy Synthetic Data Challenge
Differentially private synthetic data generation offers a recent solution to release analytically useful data while preserving the privacy of individuals in the data. In order to utilize these algorithms for public polic…
Synthetic Data GenerationLarge-Scale Public Data Improves Differentially Private Image Generation Quality
Public data has been frequently used to improve the privacy-accuracy trade-off of differentially private machine learning, but prior work largely assumes that this data come from the same distribution as the private. In …
Image Generation