Difference-Masking: Choosing What to Mask in Continued Pretraining
The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain on domain-specific data before performing some downstream task. We introduce Difference-Masking, a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. Empirically, we find that Difference-Masking outperforms baselines on continued pretraining settings across four diverse language-only and multimodal video tasks.
Code (1)
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
Self-supervised adversarial masking for 3D point cloud representation learning
Self-supervised methods have been proven effective for learning deep representations of 3D point cloud data. Although recent methods in this domain often rely on random masking of inputs, the results of this approach can…
Representation LearningFrequency Is What You Need: Word-frequency Masking Benefits Vision-Language Model Pre-training
Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of approaches: truncation, ran…
Language ModelingLanguage ModellingPOSMasking Strategies for Image Manifolds
We consider the problem of selecting an optimal mask for an image manifold, i.e., choosing a subset of the pixels of the image that preserves the manifold's geometric structure present in the original data. Such masking …
Compressive SensingEntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Transformer-based neural language models have led to breakthroughs for a variety of natural language processing (NLP) tasks. However, most models are pretrained on general domain data. We propose a methodology to produce…
NegationNegation DetectionRelationRelation Extraction+1A hemispheric two-channel code accounts for binaural unmasking in humans
Sound in noise is better detected or understood if target and masking sources originate from different locations. Mammalian physiology suggests that the neurocomputational process that underlies this binaural unmasking i…
Vocal Bursts Valence Prediction