홈 › Papers

ContextLeak: Auditing Leakage in Private In-Context Learning Methods

2025-12-18 · Jacob Choi, Shuying Cao, Xingjian Dong, Amin Banayeeanzade, Wang Bill Zhu, Robin Jia, Sai Praneeth Karimireddy arxiv

In-Context Learning (ICL) has become a standard technique for adapting Large Language Models (LLMs) to specialized tasks by supplying task-specific exemplars within the prompt. However, when these exemplars contain sensitive information, reliable privacy-preserving mechanisms are essential to prevent unintended leakage through model outputs. Many privacy-preserving methods have been proposed to protect against information leakage in this context, but there are fewer efforts on how to audit these methods. We introduce ContextLeak, the first framework to empirically measure the worst-case information leakage in ICL. ContextLeak uses canary insertion, embedding uniquely identifiable tokens in the sensitive dataset and crafting targeted queries to detect their presence. We apply ContextLeak across a range of private ICL techniques, including both heuristic prompt-based defenses and differentially private methods with formal guarantees. We show that ContextLeak reliably detects leakage across methods, and the leakage increases monotonically with the theoretical privacy budget, offering a practical signal of worst-case privacy risk. Our analysis further reveals that existing methods strike poor privacy-utility trade-offs, either completely leaking sensitive information or severely degrading performance.

📄 PDF Abstract BibTeX arXiv:2512.16059

Code (0)

등록된 구현이 없습니다.

ContextLeak: Exfiltrating LLM Agent Context via Malicious Tools

2026-08-28 · Yuqi Jia, Ruiqi Wang, Patrick Li, Yuepeng Hu 외 arxiv

Exfiltrating an LLM agent's runtime context -- such as the user prompt, execution trajectory, and tool list -- poses severe security and privacy risks to users. Such attacks can be carried out via malicious tools and typ…

Reinforcement Learning

Tight Auditing of Differentially Private Machine Learning

2023-02-15 · Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle 외

Auditing mechanisms for differential privacy use probabilistic means to empirically estimate the privacy level of an algorithm. For private machine learning, existing auditing mechanisms are tight: the empirical privacy …

Federated Learning

Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis

2025-09-22 · Joshua Ward, Xiaofeng Lin, Chi-Hua Wang, Guang Cheng arxiv

Tabular Generative Models are often argued to preserve privacy by creating synthetic datasets that resemble training data. However, auditing their empirical privacy remains challenging, as commonly used similarity metric…

Synthetic Data Evaluation

To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling

2024-11-15 · Meenatchi Sundaram Muthu Selva Annamalai, Borja Balle, Jamie Hayes, Emiliano De Cristofaro

The Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm allows the training of machine learning (ML) models with formal Differential Privacy (DP) guarantees. Since DP-SGD processes training data in batc…

Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage

2022-03-29 · CVPR 2022 1 · Zhuohang Li, Jiaxin Zhang, Luyang Liu, Jian Liu

Federated Learning (FL) framework brings privacy benefits to distributed learning systems by allowing multiple clients to participate in a learning task under the coordination of a central server without exchanging their…

Bayesian Optimization Federated Learning