Disentangling the Linguistic Competence of Privacy-Preserving BERT
Differential Privacy (DP) has been tailored to address the unique challenges of text-to-text privatization. However, text-to-text privatization is known for degrading the performance of language models when trained on perturbed text. Employing a series of interpretation techniques on the internal representations extracted from BERT trained on perturbed pre-text, we intend to disentangle at the linguistic level the distortion induced by differential privacy. Experimental results from a representational similarity analysis indicate that the overall similarity of internal representations is substantially reduced. Using probing tasks to unpack this dissimilarity, we find evidence that text-to-text privatization affects the linguistic competence across several formalisms, encoding localized properties of words while falling short at encoding the contextual relationships between spans of words.
Code (0)
등록된 구현이 없습니다.
Tasks
Privacy PreservingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence
Several studies in the literature on the interpretation of Neural Language Models (NLM) focus on the linguistic generalization abilities of pre-trained models. However, little attention is paid to how the linguistic know…
Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?
Large Language Models offer impressive language capabilities but suffer from well-known limitations, including hallucinations, biases, privacy concerns, and high computational costs. These issues are largely driven by th…
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks
Speaker anonymization systems hide the identity of speakers while preserving other information such as linguistic content and emotions. To evaluate their privacy benefits, attacks in the form of automatic speaker verific…
Language ModelingLanguage ModellingSpeaker anonymizationSpeaker VerificationHow Do BERT Embeddings Organize Linguistic Knowledge?
Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal represen…
SentenceDissociating language and thought in large language models
Large Language Models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction…