Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling
In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where the objective is altered from the reconstruction of the exact identity of randomly selected masked subwords to the prediction of their latent semantic properties. We coin the proposed pre-training technique masked latent semantic modeling (MLSM for short). In order to make the contextualized determination of the latent semantic properties of the masked subwords possible, we rely on an unsupervised technique which uses sparse coding. Our experimental results reveal that the fine-tuned performance of those models that we pre-trained via MLSM is consistently and significantly better compared to the use of vanilla MLM pretraining and other strong baselines.
Code (1)
Tasks
Language ModelingLanguage ModellingMasked Language ModelingSimilar Papers 제목 키워드 기반
Accurate and Efficient World Modeling with Masked Latent Transformers
The Dreamer algorithm has recently obtained remarkable performance across diverse environment domains by training powerful agents with simulated trajectories. However, the compressed nature of its world model's latent sp…
i-MAE: Are Latent Representations in Masked Autoencoders Linearly Separable?
Masked image modeling (MIM) has been recognized as a strong self-supervised pre-training approach in the vision domain. However, the mechanism and properties of the learned representations by such a scheme, as well as ho…
Image ReconstructionEnhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning
Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) for representation learning using the Masked Image Modeling framework. IJEPA drives representations…
image-classificationImage ClassificationRepresentation LearningGraph-level Representation Learning with Joint-Embedding Predictive Architectures
Joint-Embedding Predictive Architectures (JEPAs) have recently emerged as a novel and powerful technique for self-supervised representation learning. They aim to learn an energy-based model by predicting the latent repre…
Contrastive LearningData AugmentationGraph ClassificationGraph Regression+1Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning
Masked Image Modeling (MIM) has emerged as a promising method for deriving visual representations from unlabeled image data by predicting missing pixels from masked portions of images. It excels in region-aware learning …
Representation Learning