Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning. We create an efficient word-level adversarial attack, and use it to finetune BERT on adversarial examples generated \textit{on the fly} during training. In contrast with previous works, our method improves model robustness without using any labeled data. Experimental results show that our method improves robustness of BERT against four different word substitution-based adversarial attacks, and combining our method with adversarial training gives higher robustness than adversarial training alone. As our method improves the robustness of BERT purely with unlabeled data, it opens up the possibility of using large text datasets to train robust language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial AttackContrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Robust Pre-Training by Adversarial Contrastive Learning
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learni…
Adversarial RobustnessContrastive LearningSelf-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning…
Adversarial AttackContrastive LearningLanguage ModellingContrastive Self-Supervised Learning Leads to Higher Adversarial Susceptibility
Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. However, it is still largely unknown if the nature of the representatio…
Adversarial RobustnessSelf-Supervised LearningSensitivityVideo ClassificationCADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning
Handling out-of-distribution (OOD) samples has become a major stake in the real-world deployment of machine learning systems. This work explores the use of self-supervised contrastive learning to the simultaneous detecti…
Anomaly DetectionContrastive LearningOut-of-Distribution DetectionRobustness through Cognitive Dissociation Mitigation in Contrastive Adversarial Training
In this paper, we introduce a novel neural network training framework that increases model's adversarial robustness to adversarial attacks while maintaining high clean accuracy by combining contrastive learning (CL) with…
Adversarial RobustnessClassificationContrastive Learning