Detecting Unintended Memorization in Language-Model-Fused ASR

2022-04-20 · W. Ronny Huang, Steve Chien, Om Thakkar, Rajiv Mathews

End-to-end (E2E) models are often being accompanied by language models (LMs) via shallow fusion for boosting their overall quality as well as recognition of rare words. At the same time, several prior works show that LMs are susceptible to unintentionally memorizing rare or unique sequences in the training data. In this work, we design a framework for detecting memorization of random textual sequences (which we call canaries) in the LM training data when one has only black-box (query) access to LM-fused speech recognizer, as opposed to direct access to the LM. On a production-grade Conformer RNN-T E2E model fused with a Transformer LM, we show that detecting memorization of singly-occurring canaries from the LM training data of 300M examples is possible. Motivated to protect privacy, we also show that such memorization gets significantly reduced by per-example gradient-clipped LM training without compromising overall quality.

📄 PDF Abstract BibTeX arXiv:2204.09606

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling Memorizationmodel

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Understanding Unintended Memorization in Federated Learning

2020-06-12 · Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Françoise Beaufays

Recent works have shown that generative sequence models (e.g., language models) have a tendency to memorize rare or unique sequences in the training data. Since useful models are often trained on sensitive data, to ensur…

Clustering Federated Learning Memorization

Mitigating Unintended Memorization in Language Models via Alternating Teaching

2022-10-13 · Zhe Liu, Xuedong Zhang, Fuchun Peng

Recent research has shown that language models have a tendency to memorize rare or unique sequences in the training corpora which can thus leak sensitive attributes of user data. We employ a teacher-student framework and…

Memorization Privacy Preserving

Understanding Unintended Memorization in Language Models Under Federated Learning

2021-06-01 · NAACL (PrivateNLP) 2021 6 · Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Francoise Beaufays

Recent works have shown that language models (LMs), e.g., for next word prediction (NWP), have a tendency to memorize rare or unique sequences in the training data. Since useful LMs are often trained on sensitive data, i…

Federated Learning Memorization

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

2026-01-24 · Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl 외 arxiv

Fine-tuning Large Language Models (LLMs) on sensitive datasets carries a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII), which can violate privacy regulations and com…

How much do language models memorize?

2025-05-30 · John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan 외

We propose a new method for estimating how much a model ``knows'' about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memorization have struggled to disentangle…

Language Modeling Language Modelling Memorization