Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding
Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data. However, it's unclear whether they learn similar representations or they can be effectively combined. In this paper, we show that TAPT and ST can be complementary with simple TFS protocol by following TAPT -> Finetuning -> Self-training (TFS) process. Experimental results show that TFS protocol can effectively utilize unlabeled data to achieve strong combined gains consistently across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. We investigate various semi-supervised settings and consistently show that gains from TAPT and ST can be strongly additive by following TFS procedure. We hope that TFS could serve as an important semi-supervised baseline for future NLP studies.
Code (0)
등록된 구현이 없습니다.
Tasks
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Natural Language InferenceNatural Language UnderstandingParaphrase IdentificationSentiment AnalysisSentiment ClassificationSimilar Papers 제목 키워드 기반
SR-CIS: Self-Reflective Incremental System with Decoupled Memory and Reasoning
The ability of humans to rapidly learn new knowledge while retaining old memories poses a significant challenge for current deep learning models. To handle this challenge, we draw inspiration from human memory and learni…
Anomaly DetectionIncremental LearningM5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse da…
Contrastive LearningHandling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labeling
In this paper, we address the problem of effectively self-training neural networks in a low-resource setting. Self-training is frequently used to automatically increase the amount of training data. However, in a low-reso…
Auxiliary LearningChunkingNERCoMA: Complementary Masking and Hierarchical Dynamic Multi-Window Self-Attention in a Unified Pre-training Framework
Masked Autoencoders (MAE) achieve self-supervised learning of image representations by randomly removing a portion of visual tokens and reconstructing the original image as a pretext task, thereby significantly enhancing…
Self-Supervised LearningSelf-Training and Adversarial Background Regularization for Unsupervised Domain Adaptive One-Stage Object Detection
Deep learning-based object detectors have shown remarkable improvements. However, supervised learning-based methods perform poorly when the train data and the test data have different distributions. To address the issue,…
Domain AdaptationObjectobject-detectionObject Detection+1