Adversarial Self-Attention for Language Understanding
Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances the self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose \textit{Adversarial Self-Attention} mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct a comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gains compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.
Code (1)
Tasks
Machine Reading ComprehensionNamed Entity Recognition (NER)Natural Language InferenceParaphrase IdentificationSemantic SimilaritySemantic Textual SimilaritySentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Alignment Attention by Matching Key and Query Distributions
The neural attention mechanism has been incorporated into deep neural networks to achieve state-of-the-art performance in various domains. Most such models use multi-head self-attention which is appealing for the ability…
Graph AttentionQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Fooling Vision and Language Models Despite Localization and Attention Mechanism
Adversarial attacks are known to succeed on classifiers, but it has been an open question whether more complex vision systems are vulnerable. In this paper, we study adversarial examples for vision and language models, w…
Dense CaptioningNatural Language UnderstandingOpen-Ended Question AnsweringQuestion Answering+2Enhancing Essay Scoring with Adversarial Weights Perturbation and Metric-specific AttentionPooling
The objective of this study is to improve automated feedback tools designed for English Language Learners (ELLs) through the utilization of data science techniques encompassing machine learning, natural language processi…
Automated Essay ScoringLanguage ModellingNatural Language UnderstandingSelf-Supervised LearningSALSA-TEXT : self attentive latent space based adversarial text generation
Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based architectures to improve the performance …
Adversarial TextImage GenerationSentenceSentence Compression+1Towards Efficient Adversarial Training on Vision Transformers
Vision Transformer (ViT), as a powerful alternative to Convolutional Neural Network (CNN), has received much attention. Recent work showed that ViTs are also vulnerable to adversarial examples like CNNs. To build robust …