NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents
While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechanism. To address this, we propose NextLevelBERT, a Masked Language Model operating not on tokens, but on higher-level semantic representations in the form of text embeddings. We pretrain NextLevelBERT to predict the vector representation of entire masked text chunks and evaluate the effectiveness of the resulting document vectors on three types of tasks: 1) Semantic Textual Similarity via zero-shot document embeddings, 2) Long document classification, 3) Multiple-choice question answering. We find that next-level Masked Language Modeling is an effective technique to tackle long-document use cases and can outperfor much larger embedding models as long as the required level of detail of semantic information is not too fine. Our models and code are publicly available online.
Code (1)
Tasks
Document ClassificationLanguage ModelingLanguage ModellingMasked Language ModelingMultiple-choiceQuestion AnsweringSemantic Textual SimilaritySimilar Papers 제목 키워드 기반
MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection
Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher…
AttributeLanguage ModelingLanguage ModellingMasked Language Modeling+1LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models
Vision-language pre-training like CLIP has shown promising performance on various downstream tasks such as zero-shot image classification and image-text retrieval. Most of the existing CLIP-alike works usually adopt rela…
image-classificationImage ClassificationImage-text RetrievalLanguage Modeling+4MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning
Multimodal representation learning has shown promising improvements on various vision-language tasks. Most existing methods excel at building global-level alignment between vision and language while lacking effective fin…
Image-text Retrievalmultimodal interactionQuestion AnsweringRepresentation Learning+6HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation
Masked image modeling (MIM) with transformer backbones has recently been exploited as a powerful self-supervised pre-training technique. The existing MIM methods adopt the strategy to mask random patches of the image and…
Contrastive LearningImage SegmentationMedical Image SegmentationSelf-Supervised Learning+1MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation
Sign language generation (SLG) aims to translate written texts into expressive sign motions, bridging communication barriers for the Deaf and Hard-of-Hearing communities. Recent studies formulate SLG within the language …