How does the task complexity of masked pretraining objectives affect downstream performance?
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked token, have recently shown comparable results to MLM, no objectives with a masking scheme actually outperform it in downstream tasks. Motivated by the assumption that their lack of complexity plays a vital role in the degradation, we validate whether more complex masked objectives can achieve better results and investigate how much complexity they should have to perform comparably to MLM. Our results using GLUE, SQuAD, and Universal Dependencies benchmarks demonstrate that more complicated objectives tend to show better downstream results with at least half of the MLM complexity needed to perform comparably to MLM. Finally, we discuss how we should pretrain a model using a masked objective from the task complexity perspective.
Code (1)
Tasks
Language ModelingLanguage ModellingMasked Language ModelingSimilar Papers 제목 키워드 기반
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling
Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to predict a random sample of input tokens that…
Language ModelingLanguage ModellingMasked Language ModelingSentenceA Unified Framework for In-Context Learning with Causal and Masked Language Models
In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction…
A Thorough Evaluation of Task-Specific Pretraining for Summarization
Task-agnostic pretraining objectives like masked language models or corrupted span prediction are applicable to a wide range of NLP downstream tasks (Raffel et al.,2019), but are outperformed by task-specific pretraining…
ProteinJEPA: Latent prediction complements protein language models
Protein language models are trained primarily with masked language modeling (MLM), which predicts amino-acid identities at masked positions. We ask whether latent-space prediction can complement these token-level objecti…
The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
We analyze the masked language modeling pretraining objective function from the perspective of the distributional hypothesis. We investigate whether better sample efficiency and the better generalization capability of mo…
Language ModelingLanguage ModellingMasked Language ModelingSemantic Similarity+1