Learning Helpful Inductive Biases from Self-Supervised Pretraining
Large pretrained language models demonstrate strong, language-specific biases during fine-tuning that allow them to solve language tasks better than models without pretraining. We aim to characterize these biases, and identify the amount of pretraining that is necessary to acquire them. We introduce a new English language diagnostic set called MSGS (Mixed Signals Generalization Set) which contains two types of data: mixed data in which the labels are consistent with both a linguistic classification (e.g., Is the main verb in the progressive form?) and a superficial surface one (e.g., Does "the" precede "a"?); and unmixed data in which the labels align only with the linguistic feature. We fine-tune RoBERTa on mixed data (with and without small amounts of inoculating unmixed data) and test on unmixed data to see which feature it has bias in favor of. We pretrain RoBERTa from scratch on quantities of data ranging from 1M to 1B words and compare their performance on MSGS to the publicly available RoBERTa-Base. We find steady growth in linguistic bias with increased pretraining data. The models we test can usually represent the linguistic features, but they only learn to prefer to generalize based on these features with significant pretraining. In the absence of inoculating data, only RoBERTa-Base consistently demonstrates a linguistic bias with any regularity.
Code (0)
등록된 구현이 없습니다.
Tasks
DiagnosticSimilar Papers 제목 키워드 기반
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
One reason pretraining on self-supervised linguistic tasks is effective is that it teaches models features that are helpful for language understanding. However, we want pretrained models to learn not only to represent li…
Binary ClassificationDiagnosticPrior Knowledge-Guided Attention in Self-Supervised Vision Transformers
Recent trends in self-supervised representation learning have focused on removing inductive biases from training pipelines. However, inductive biases can be useful in settings when limited data are available or provide a…
Heart SegmentationMedical Image AnalysisRepresentation LearningA Theoretical Study of Inductive Biases in Contrastive Learning
Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi e…
ClusteringContrastive LearningSelf-Supervised LearningYou Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly charact…
Self-Supervised LearningRepresentation LearningWhat do Large Language Models Learn beyond Language?
Large language models (LMs) have rapidly become a mainstay in Natural Language Processing. These models are known to acquire rich linguistic knowledge from training on large amounts of text. In this paper, we investigate…
Inductive Learning