Natural Language Understanding with Privacy-Preserving BERT
Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingNatural Language UnderstandingPrivacy PreservingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices
The field of Natural Language Processing (NLP) is currently undergoing a revolutionary transformation driven by the power of pre-trained Large Language Models (LLMs) based on groundbreaking Transformer architectures. As …
CPULanguage ModellingPrivacy PreservingDifferential Privacy for Text Analytics via Natural Text Sanitization
Texts convey sophisticated knowledge. However, texts also convey sensitive information. Despite the success of general-purpose language models and domain-specific mechanisms with differential privacy (DP), existing text …
Language ModelingLanguage ModellingPrivacy PreservingSensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
Massive digital data processing provides a wide range of opportunities and benefits, but at the cost of endangering personal data privacy. Anonymisation consists in removing or replacing sensitive information from data, …
Feature EngineeringGeneral ClassificationPrivacy Preserving In-Context-Learning Framework for Large Language Models
Large language models (LLMs) have significantly transformed natural language understanding and generation, but they raise privacy concerns due to potential exposure of sensitive information. Studies have highlighted the …
Natural Language UnderstandingText GenerationPrivacy-Preserving Language Model Inference with Instance Obfuscation
Language Models as a Service (LMaaS) offers convenient access for developers and researchers to perform inference using pre-trained language models. Nonetheless, the input data and the inference results containing privat…
BenchmarkingLanguage ModelingLanguage Modellingmodel+2