Learning Data Augmentation Schedules for Natural Language Processing
Despite its proven efficiency in other fields, data augmentation is less popular in the context of natural language processing (NLP) due to its complexity and limited results. A recent study (Longpre et al., 2020) showed for example that task-agnostic data augmentations fail to consistently boost the performance of pretrained transformers even in low data regimes. In this paper, we investigate whether data-driven augmentation scheduling and the integration of a wider set of transformations can lead to improved performance where fixed and limited policies were unsuccessful. Our results suggest that, while this approach can help the training process in some settings, the improvements are unsubstantial. This negative result is meant to help researchers better understand the limitations of data augmentation for NLP.
Code (1)
Tasks
Data AugmentationSchedulingSimilar Papers 제목 키워드 기반
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations. Properly chosen augmentation policies can lead t…
Data AugmentationImage AugmentationLINDA: Unsupervised Learning to Interpolate in Natural Language Processing
Despite the success of mixup in data augmentation, its applicability to natural language processing (NLP) tasks has been limited due to the discrete and variable-length nature of natural languages. Recent studies have th…
Data Augmentationtext-classificationText ClassificationLINDA: Unsupervised Learning to Interpolate in Natural Language Processing
Despite the success of mixup in data augmentation, its applicability to natural language processing (NLP) tasks has been limited due to the discrete and variable-length nature of natural languages. Recent studies have th…
Data Augmentationtext-classificationText ClassificationInvestigating Masking-based Data Generation in Language Models
The current era of natural language processing (NLP) has been defined by the prominence of pre-trained language models since the advent of BERT. A feature of BERT and models with similar architecture is the objective of …
Data AugmentationLanguage ModelingLanguage ModellingMasked Language ModelingHow Data Augmentation affects Optimization for Linear Regression
Though data augmentation has rapidly emerged as a key tool for optimization in modern machine learning, a clear picture of how augmentation schedules affect optimization and interact with optimization hyperparameters suc…
Data AugmentationregressionStochastic Optimization