Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Generative power of a protein language model trained on multiple sequence alignments
Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families and learn constraints associated to protein structure and function. They thus op…
Language ModelingLanguage ModellingMasked Language ModelingProtein Design+1What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pre…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+1Data Augmentation for Biomedical Factoid Question Answering
We study the effect of seven data augmentation (da) methods in factoid question answering, focusing on the biomedical domain, where obtaining training instances is particularly difficult. We experiment with data from the…
Data AugmentationInformation RetrievalLanguage ModelingLanguage Modelling+8Contextual Representation Learning beyond Masked Language Modeling
How do masked language models (MLMs) such as BERT learn contextual representations? In this work, we analyze the learning dynamics of MLMs. We find that MLMs adopt sampled embeddings as anchors to estimate and inject con…
Language ModelingLanguage ModellingMasked Language ModelingRepresentation LearningSecureBERT: A Domain-Specific Language Model for Cybersecurity
Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the wor…
Language ModelingLanguage ModellingMasked Language Modelingmodel+1POS-BERT: Point Cloud One-Stage BERT Pre-Training
Recently, the pre-training paradigm combining Transformer and masked language modeling has achieved tremendous success in NLP, images, and point clouds, such as BERT. However, directly extending BERT from NLP to point cl…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning framework, we introduce two pre-training t…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderLanguage Modeling+5LinkBERT: Pretraining Language Models with Document Links
Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that s…
Document ClassificationLanguage ModelingLanguage ModellingMasked Language Modeling+10Token Dropping for Efficient BERT Pretraining
Transformer-based models generally allocate the same amount of computation for each token in a given sequence. We develop a simple but effective "token dropping" method to accelerate the pretraining of transformer models…
Language ModelingLanguage ModellingMasked Language ModelingCan Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining?
Identifying argument components from unstructured texts and predicting the relationships expressed among them are two primary steps of argument mining. The intrinsic complexity of these tasks demands powerful learning mo…
Argument MiningLanguage ModelingLanguage ModellingMasked Language Modeling+2What to Hide from Your Students: Attention-Guided Masked Image Modeling
Transformers and masked language modeling are quickly being adopted and explored in computer vision as vision transformers and masked image modeling (MIM). In this work, we argue that image token masking differs from tok…
Language ModelingLanguage ModellingMasked Language ModelingSelf-Supervised LearningHOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation
Pre-training has been adopted in a few of recent works for Vision-and-Language Navigation (VLN). However, previous pre-training methods for VLN either lack the ability to predict future actions or ignore the trajectory c…
Decision MakingLanguage ModelingLanguage ModellingMasked Language Modeling+3How does the pre-training objective affect what large language models learn about linguistic properties?
Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our k…
Language ModelingLanguage ModellingMasked Language ModelingGeographic Adaptation of Pretrained Language Models
While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain…
Language IdentificationLanguage ModelingLanguage ModellingMasked Language Modeling+2SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understanding
Prevailing deep models are single-purpose and overspecialize at individual tasks. However, when being extended to new tasks, they typically forget previously learned skills and learn from scratch. We address this issue b…
Language ModellingMasked Language ModelingMixture-of-ExpertsMulti-Task Learning+1"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction
Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The…
Grammatical Error CorrectionLanguage ModelingLanguage ModellingMasked Language Modeling+1Probing BERT's priors with serial reproduction chains
Sampling is a promising bottom-up method for exposing what generative models have learned about language, but it remains unclear how to generate representative samples from popular masked language models (MLMs) like BERT…
Language ModellingMasked Language ModelingVU-BERT: A Unified framework for Visual Dialog
The visual dialog task attempts to train an agent to answer multi-turn questions given an image, which requires the deep understanding of interactions between the image and dialog history. Existing researches tend to emp…
Language ModelingLanguage ModellingMasked Language ModelingRetrieval+1Transformer Quality in Linear Time
We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker sin…
8kLanguage ModelingLanguage ModellingMasked Language ModelingShould You Mask 15% in Masked Language Modeling?
Masked language models (MLMs) conventionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations; this masking rate has been widely used, regardless of mo…
Language ModelingLanguage ModellingMasked Language Modeling