On Masked Pre-training and the Marginal Likelihood
Masked pre-training removes random input dimensions and learns a model that can predict the missing values. Empirical results indicate that this intuitive form of self-supervised learning yields models that generalize very well to new domains. A theoretical understanding is, however, lacking. This paper shows that masked pre-training with a suitable cumulative scoring function corresponds to maximizing the model's marginal likelihood, which is de facto the Bayesian model selection measure of generalization. Beyond shedding light on the success of masked pre-training, this insight also suggests that Bayesian models can be trained with appropriately designed self-supervision. Empirically, we confirm the developed theory and explore the main learning principles of masked pre-training in large language models.
Code (1)
Similar Papers 제목 키워드 기반
Marginal Post Processing of Bayesian Inference Products with Normalizing Flows and Kernel Density Estimators
Bayesian analysis has become an indispensable tool across many different cosmological fields including the study of gravitational waves, the Cosmic Microwave Background and the 21-cm signal from the Cosmic Dawn among oth…
Bayesian InferenceExperimental DesignDiscrete Tilt Matching
Masked diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. While reinforcement learning (RL) methods have recently been adapted to dLLM fine-tuning, their objectives typicall…
Reinforcement LearningScalable Marginal Likelihood Estimation for Model Selection in Deep Learning
Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In t…
Deep Learningimage-classificationImage ClassificationModel Selection+2Last Layer Marginal Likelihood for Invariance Learning
Data augmentation is often used to incorporate inductive biases into models. Traditionally, these are hand-crafted and tuned with cross validation. The Bayesian paradigm for model selection provides a path towards end-to…
Data AugmentationGaussian ProcessesModel SelectionOn the marginal likelihood and cross-validation
In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typicall…
scoring rule