Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining
Audio can disclose PII, particularly when combined with related text data. Therefore, it is essential to develop tools to detect privacy leakage in Contrastive Language-Audio Pretraining(CLAP). Existing MIAs need audio as input, risking exposure of voiceprint and requiring costly shadow models. We first propose PRMID, a membership inference detector based probability ranking given by CLAP, which does not require training shadow models but still requires both audio and text of the individual as input. To address these limitations, we then propose USMID, a textual unimodal speaker-level membership inference detector, querying the target model using only text data. We randomly generate textual gibberish that are clearly not in training dataset. Then we extract feature vectors from these texts using the CLAP model and train a set of anomaly detectors on them. During inference, the feature vector of each test text is input into the anomaly detector to determine if the speaker is in the training set (anomalous) or not (normal). If available, USMID can further enhance detection by integrating real audio of the tested speaker. Extensive experiments on various CLAP model architectures and datasets demonstrate that USMID outperforms baseline methods using only text data.
Code (0)
등록된 구현이 없습니다.
Tasks
AllMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Membership Inference for Contrastive Pre-training Models with Text-only PII Queries
Contrastive pretraining models such as CLIP and CLAP, serve as the ubiquitous perceptual backbones for modern multimodal large models, yet their reliance on web-scale data raises growing concerns about memorizing Persona…
Quantifying and Mitigating Privacy Risks of Contrastive Learning
Data is the key factor to drive the development of machine learning (ML) during the past decade. However, high-quality data, in particular labeled data, is often hard and expensive to collect. To leverage large-scale unl…
AttributeBIG-bench Machine LearningContrastive LearningData Augmentation+2Membership Inference Attacks Against Vision-Language Models
Vision-Language Models (VLMs), built on pre-trained vision encoders and large language models (LLMs), have shown exceptional multi-modal understanding and dialog capabilities, positioning them as catalysts for the next t…
Inference AttackMembership Inference AttackCLMIA: Membership Inference Attacks via Unsupervised Contrastive Learning
Since machine learning model is often trained on a limited data set, the model is trained multiple times on the same data sample, which causes the model to memorize most of the training set data. Membership Inference Att…
Contrastive LearningLearning to Detect Language Model Training Data via Active Reconstruction
Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likelihoods or text generations. In this work…
Reinforcement Learning