Sequence-Level Leakage Risk of Training Data in Large Language Models

2024-12-15 · Trishita Tiwari, G. Edward Suh

This work quantifies the risk of training data leakage from LLMs (Large Language Models) using sequence-level probabilities. Computing extraction probabilities for individual sequences provides finer-grained information than has been studied in prior benchmarking work. We re-analyze the effects of decoding schemes, model sizes, prefix lengths, partial sequence leakages, and token positions to uncover new insights that were not possible in previous works due to their choice of metrics. We perform this study on two pre-trained models, Llama and OPT, trained on the Common Crawl and The Pile respectively. We discover that 1) Extraction Rate, the predominant metric used in prior quantification work, underestimates the threat of leakage of training data in randomized LLMs by as much as 2.14X. 2) Although on average, larger models and longer prefixes can extract more data, this is not true for a substantial portion of individual sequences. 30.4-41.5% of our sequences are easier to extract with either shorter prefixes or smaller models. 3) Contrary to previous beliefs, partial leakage in commonly used decoding schemes like top-k and top-p is not easier than leaking verbatim training data. The aim of this work is to encourage the adoption of this metric for future work on quantification of training data extraction.

📄 PDF Abstract BibTeX arXiv:2412.11302

Code (0)

등록된 구현이 없습니다.

Tasks

Benchmarking

Methods 이 논문이 사용한 방법론

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

OPT OPT is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. The model uses an AdamW optimizer and weight decay of 0.1. It follows a linear…

Analyzing Leakage of Personally Identifiable Information in Language Models

2023-02-01 · Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople 외

Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Informat…

Sentence

(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs

2025-10-07 · Jiashu Tao, Reza Shokri arxiv

Machine learning models are known to leak sensitive information, as they inevitably memorize (parts of) their training data. More alarmingly, large language models (LLMs) are now trained on nearly all available data, whi…

Computational Efficiency

Hidden Leaks in Time Series Forecasting: How Data Leakage Affects LSTM Evaluation Across Configurations and Validation Strategies

2025-12-07 · Salma Albelali, Moataz Ahmed arxiv

Deep learning models, particularly Long Short-Term Memory (LSTM) networks, are widely used in time series forecasting due to their ability to capture complex temporal dependencies. However, evaluation integrity is often …

Time Series Forecasting

Prior Aware Memorization: An Efficient Metric for Distinguishing Memorization from Generalization in Large Language Models

2026-02-21 · Trishita Tiwari, Ari Trachtenberg, G. Edward Suh arxiv

Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance. A central challenge in assessing this risk is distinguishing genuine memorization of…

MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agents

2026-05-29 · Alexander Gurung, Spandana Gella, Alexandre Drouin, Issam H. Laradji 외 arxiv

Deep research agents increasingly combine private local documents with external tools like web retrieval, creating a privacy risk: an agent's external queries may leak sensitive information from its local context. This r…

Reinforcement Learning