Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Do Transformers Parse while Predicting the Masked Word?
Pre-trained language models have been shown to encode linguistic structures, e.g. dependency and constituency parse trees, in their embeddings while being trained on unsupervised loss functions like masked language model…
Constituency ParsingLanguage ModelingLanguage ModellingMasked Language ModelingGenerating multiple-choice questions for medical question answering with distractors and cue-masking
Medical multiple-choice question answering (MCQA) is particularly difficult. Questions may describe patient symptoms and ask for the correct diagnosis, which requires domain knowledge and complex reasoning. Standard lang…
Language ModelingLanguage ModellingMasked Language ModelingMedical Question Answering+4StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training
In this paper, we present StrucTexTv2, an effective document image pre-training framework, by performing masked visual-textual prediction. It consists of two self-supervised pre-training tasks: masked image modeling and …
Document Image Classificationimage-classificationImage ClassificationLanguage Modeling+4Domain-adapted large language models for classifying nuclear medicine reports
With the growing use of transformer-based language models in medicine, it is unclear how well these models generalize to nuclear medicine which has domain-specific vocabulary and unique reporting styles. In this study, w…
Domain AdaptationLanguage ModelingLanguage ModellingMasked Language ModelingWeighted Sampling for Masked Language Modeling
Masked Language Modeling (MLM) is widely used to pretrain language models. The standard random masking strategy in MLM causes the pre-trained language models (PLMs) to be biased toward high-frequency tokens. Representati…
Language ModelingLanguage ModellingMasked Language ModelingPrompt Learning+6Efficient Masked Autoencoders with Self-Consistency
Inspired by the masked language modeling (MLM) in natural language processing tasks, the masked image modeling (MIM) has been recognized as a strong self-supervised pre-training method in computer vision. However, the hi…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+4Symbolic Discovery of Optimization Algorithms
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite an…
Contrastive Learningimage-classificationImage ClassificationLanguage Modeling+2Capturing Topic Framing via Masked Language Modeling
Differential framing of issues can lead to divergent world views on important issues. This is especially true in domains where the information presented can reach a large audience, such as traditional and social media. S…
ArticlesLanguage ModelingLanguage ModellingMasked Language ModelingRepresentation Deficiency in Masked Language Modeling
Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt…
Language ModelingLanguage ModellingMasked Language ModelingTagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrieval
Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years. Most of the existing methods either transfer the knowledge of image-text pretraining model to video-text retrieval t…
Language ModelingLanguage ModellingMasked Language ModelingRetrieval+5Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers
Bidirectional Transformer architectures are state-of-the-art sequential recommendation models that use a bi-directional representation capacity based on the Cloze task, a.k.a. Masked Language Modeling. The latter aims to…
Language ModelingLanguage ModellingMasked Language ModelingSequential RecommendationA Cohesive Distillation Architecture for Neural Language Models
A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size, which prevents research groups without a necessary hardware infrastructure from participating in the development proces…
Knowledge DistillationLanguage ModelingLanguage ModellingMasked Language Modeling+2Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves excellent transfer performance on both vi…
Cross-Modal RetrievalImage Captioningimage-classificationImage Classification+10Cramming: Training a Language Model on a Single GPU in One Day
Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While …
GPULanguage ModelingLanguage ModellingMasked Language ModelingMicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning
Transformer language models (TLMs) are critical for most NLP tasks, but they are difficult to create for low-resource languages because of how much pretraining data they require. In this work, we investigate two techniqu…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language Modeling+2Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Models
With increasing scale, large language models demonstrate both quantitative improvement and new qualitative capabilities, especially as zero-shot learners, like GPT-3. However, these results rely heavily on delicate promp…
Language ModelingLanguage ModellingMasked Language ModelingSelf-Supervised Learning+1APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
Logical reasoning of text is an important ability that requires understanding the information present in the text, their interconnections, and then reasoning through them to infer new conclusions. Prior works on improvin…
Data AugmentationLanguage ModelingLanguage ModellingLogical Reasoning+2Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AS…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)automatic-speech-translationDecoder+8Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking
Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM…
Language ModelingLanguage ModellingMasked Language ModelingUniform Masking Prevails in Vision-Language Pretraining
Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining. To implement MLM, the researcher must make two design choices: the masking strategy, which determines which toke…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+1