Training Data Extraction From Pre-trained Language Models: A Survey
As the deployment of pre-trained language models (PLMs) expands, pressing security concerns have arisen regarding the potential for malicious extraction of training data, posing a threat to data privacy. This study is the first to provide a comprehensive survey of training data extraction from PLMs. Our review covers more than 100 key papers in fields such as natural language processing and security. First, preliminary knowledge is recapped and a taxonomy of various definitions of memorization is presented. The approaches for attack and defense are then systemized. Furthermore, the empirical findings of several quantitative studies are highlighted. Finally, future research directions based on this review are suggested.
Code (0)
등록된 구현이 없습니다.
Tasks
MemorizationSurveySimilar Papers 제목 키워드 기반
VLP: A Survey on Vision-Language Pre-training
In the past few years, the emergence of pre-training models has brought uni-modal fields such as computer vision (CV) and natural language processing (NLP) to a new era. Substantial works have shown they are beneficial f…
SurveyA Survey on Open Information Extraction from Rule-based Model to Large Language Model
Open Information Extraction (OpenIE) represents a crucial NLP task aimed at deriving structured information from unstructured text, unrestricted by relation type or domain. This survey paper provides an overview of OpenI…
Language ModelingLanguage ModellingLarge Language Modelmodel+1Pre-Trained Language Models for Keyphrase Prediction: A Review
Keyphrase Prediction (KP) is essential for identifying keyphrases in a document that can summarize its content. However, recent Natural Language Processing (NLP) advances have developed more efficient KP models using dee…
Keyphrase ExtractionKeyphrase GenerationPredictionEvent Extraction: A Survey
Extracting the reported events from text is one of the key research themes in natural language processing. This process includes several tasks such as event detection, argument extraction, role labeling. As one of the mo…
Event DetectionEvent ExtractionNatural Language UnderstandingSurveyA Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers
Relation extraction (RE) involves identifying the relations between entities from underlying content. RE serves as the foundation for many natural language processing (NLP) and information retrieval applications, such as…
Information RetrievalKnowledge Graph CompletionQuestion AnsweringRelation+4