Perturbation-based Self-supervised Attention for Text Classification
For text classification, the traditional attention mechanism usually pays too much attention to words that appear frequently and needs a lot of labeled data for learning a good distribution. Introducing human attention is a classical method, but it needs a high cost of manual labeling. This paper proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overheads. Specifically, we add as much noise as possible to all the words in the sentence simultaneously while without changing their semantics and predictions. According to the words that tolerate more noise are supposed to be less significant, we can obtain attention supervision information and utilize it to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly promote the performance of current attention-based models and is more effective than existing self-supervised methods. We also provide visualization analysis to verify the effectiveness of our approach.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationSentencetext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
In text classification, the traditional attention mechanisms usually focus too much on frequent words, and need extensive labeled data in order to learn. This paper proposes a perturbation-based self-supervised attention…
Sentencetext-classificationText ClassificationMixture of Self-Supervised Learning
Self-supervised learning is popular method because of its ability to learn features in images without using its labels and is able to overcome limited labeled datasets used in supervised learning. Self-supervised learnin…
Classificationimage-classificationImage ClassificationSelf-Supervised LearningAugmented Contrastive Self-Supervised Learning for Audio Invariant Representations
Improving generalization is a major challenge in audio classification due to labeled data scarcity. Self-supervised learning (SSL) methods tackle this by leveraging unlabeled data to learn useful features for downstream …
Audio ClassificationClassificationContrastive LearningSelf-Supervised LearningGraph Adversarial Self-Supervised Learning
This paper studies a long-standing problem of learning the representations of a whole graph without human supervision. The recent self-supervised learning methods train models to be invariant to the transformations (view…
Graph ClassificationSelf-Supervised LearningAdversarial Training Methods for Semi-Supervised Text Classification
Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both metho…
ClassificationGeneral ClassificationSemi-Supervised Text ClassificationSentiment Analysis+2