Adversarial Training with Contrastive Learning in NLP
For years, adversarial training has been extensively studied in natural language processing (NLP) settings. The main goal is to make models robust so that similar inputs derive in semantically similar outcomes, which is not a trivial problem since there is no objective measure of semantic similarity in language. Previous works use an external pre-trained NLP model to tackle this challenge, introducing an extra training stage with huge memory consumption during training. However, the recent popular approach of contrastive learning in language processing hints a convenient way of obtaining such similarity restrictions. The main advantage of the contrastive learning approach is that it aims for similar data points to be mapped close to each other and further from different ones in the representation space. In this work, we propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task using the benefits of contrastive learning. The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning. In NLP experiments, we applied ATCL to language modeling and neural machine translation tasks. The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks without using a pre-trained model.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingMachine TranslationSemantic SimilaritySemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PointACL:Adversarial Contrastive Learning for Robust Point Clouds Representation under Adversarial Attack
Despite recent success of self-supervised based contrastive learning model for 3D point clouds representation, the adversarial robustness of such pre-trained models raised concerns. Adversarial contrastive learning (ACL)…
3D ClassificationAdversarial AttackAdversarial RobustnessContrastive LearningSimple Contrastive Representation Adversarial Learning for NLP Tasks
Self-supervised learning approach like contrastive learning is attached great attention in natural language processing. It uses pairs of training data augmentations to build a classification task for an encoder with well…
Contrastive LearningNatural Language UnderstandingSelf-Supervised LearningSemantic Textual Similarity+2Robust Pre-Training by Adversarial Contrastive Learning
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learni…
Adversarial RobustnessContrastive LearningRobustness through Cognitive Dissociation Mitigation in Contrastive Adversarial Training
In this paper, we introduce a novel neural network training framework that increases model's adversarial robustness to adversarial attacks while maintaining high clean accuracy by combining contrastive learning (CL) with…
Adversarial RobustnessClassificationContrastive LearningAdversarial Training using Contrastive Divergence
To protect the security of machine learning models against adversarial examples, adversarial training becomes the most popular and powerful strategy against various adversarial attacks by injecting adversarial examples i…