How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?
Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situationally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any particular task), Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrichet al., 2015), we conduct a systematic examination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XLNet, and RoBERTa. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may confer improvements.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classification
Recent advancements in AI have democratized its deployment as a healthcare assistant. While pretrained models from large-scale visual and audio datasets have demonstrably generalized to this task, surprisingly, no studie…
Data AugmentationSound ClassificationTAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Models
Existing foundation models (FMs) in the medical domain often require extensive fine-tuning or rely on training resource-intensive decoders, while many existing encoders are pretrained with objectives biased toward specif…
The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasound
Data augmentation is a central component of joint embedding self-supervised learning (SSL). Approaches that work for natural images may not always be effective in medical imaging tasks. This study systematically investig…
ClassificationData AugmentationDiagnosticLine Detection+1Learning Data Augmentation Schedules for Natural Language Processing
Despite its proven efficiency in other fields, data augmentation is less popular in the context of natural language processing (NLP) due to its complexity and limited results. A recent study (Longpre et al., 2020) showed…
Data AugmentationSchedulingTransfer Learning with EfficientNet for Accurate Leukemia Cell Classification
Accurate classification of Acute Lymphoblastic Leukemia (ALL) from peripheral blood smear images is essential for early diagnosis and effective treatment planning. This study investigates the use of transfer learning wit…
Transfer LearningData Augmentation