Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

2023-10-27 · Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, Yejin Choi

The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.

📄 PDF Abstract BibTeX arXiv:2310.17884

Code (1)

skywalker023/confaide pytorch

Tasks

Privacy Preserving

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Multi-Head Attention 설명 없음

Attention 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Towards eliciting latent knowledge from LLMs with mechanistic interpretability

2025-05-20 · Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda

As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their o…

PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

2025-12-31 · Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An 외 arxiv

Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk:…

Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective

2026-04-20 · Meifang Chen, Zhe Yang, Huang Nianchen, Yizhan Huang 외 arxiv

Code secrets are sensitive assets for software developers, and their leakage poses significant cybersecurity risks. While the rapid development of AI code assistants powered by Code Large Language Models (CLLMs), CLLMs a…

Vicious Classifiers: Assessing Inference-time Data Reconstruction Risk in Edge Computing

2022-12-08 · Mohammad Malekzadeh, Deniz Gunduz

Privacy-preserving inference in edge computing paradigms encourages the users of machine-learning services to locally run a model on their private input and only share the models outputs for a target task with the server…

Edge-computing Privacy Preserving Reconstruction Attack

Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing

2026-05-11 · Ari Holtzman, Peter West arxiv

Language models are deployed in settings that require compartmentalization: system prompts should not be disclosed, chain-of-thought reasoning is hidden from users, and sensitive data passes through shared contexts. We t…