Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE
This technical report briefly describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the widely used general language understanding evaluation (GLUE) benchmar…
Common Sense Reasoningcoreference-resolutionCoreference ResolutionLanguage Modeling+5Global memory transformer for processing long documents
Transformer variants dominate the state-of-the-art in different natural language processing tasks such as translation, reading comprehension and summarization. Our paper is more directed to use general memory slots added…
Language ModelingLanguage ModellingMasked Language ModelingReading ComprehensionNonparametric Masked Language Modeling
Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that r…
Language ModelingLanguage ModellingMasked Language ModelingQuestion Answering+1Comparison Study Between Token Classification and Sequence Classification In Text Classification
Unsupervised Machine Learning techniques have been applied to Natural Language Processing tasks and surpasses the benchmarks such as GLUE with great success. Building language models approach achieves good results in one…
ClassificationLanguage ModelingLanguage ModellingMasked Language Modeling+4Self-supervised vision-language pretraining for Medical visual question answering
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To…
Contrastive LearningImage-text matchingLanguage ModelingLanguage Modelling+6Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning
Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language mo…
cross-modal alignmentImage-text RetrievalLanguage ModelingLanguage Modelling+8Unified Multimodal Model with Unlikelihood Training for Visual Dialog
The task of visual dialog requires a multimodal chatbot to answer sequential questions from humans about image content. Prior work performs the standard likelihood training for answer generation on the positive instances…
Answer GenerationChatbotLanguage ModelingLanguage Modelling+3Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning
Social media has become an important information source for crisis management and provides quick access to ongoing developments and critical information. However, classification models suffer from event-related biases an…
Hierarchical Multi-label ClassificationLanguage ModelingLanguage ModellingManagement+4Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge
Contextual synonym knowledge is crucial for those similarity-oriented tasks whose core challenge lies in capturing semantic similarity between entities in their contexts, such as entity linking and entity matching. Howev…
Entity LinkingLanguage ModelingLanguage ModellingMasked Language Modeling+3Leveraging per Image-Token Consistency for Vision-Language Pre-training
Most existing vision-language pre-training (VLP) approaches adopt cross-modal masked language modeling (CMLM) to learn vision-language associations. However, we find that CMLM is insufficient for this purpose according t…
Language ModelingLanguage ModellingMasked Language ModelingSentenceReduce, Reuse, Recycle: Improving Training Efficiency with Distillation
Methods for improving the efficiency of deep network training (i.e. the resources required to achieve a given level of model quality) are of immediate benefit to deep learning practitioners. Distillation is typically use…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1HanTrans: An Empirical Study on Cross-Era Transferability of Chinese Pre-trained Language Model
The pre-trained language model has recently dominated most downstream tasks in the NLP area. Particularly, bidirectional Encoder Representations from Transformers (BERT) is the most iconic pre-trained language model amon…
Language ModelingLanguage ModellingMasked Language ModelingCodeEditor: Learning to Edit Source Code with Pre-trained Models
Developers often perform repetitive code editing activities for various reasons (e.g., code refactoring) during software development. Pre-trained code editing models have achieved the state-of-the-art (SOTA) results. Pre…
Language ModelingLanguage ModellingMasked Language ModelingLeveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion
Detecting emotions expressed in text has become critical to a range of fields. In this work, we investigate ways to exploit label correlations in multi-label emotion recognition models to improve emotion detection. First…
Emotion RecognitionLanguage ModelingLanguage ModellingMasked Language ModelingRetrieval Oriented Masking Pre-training Language Model for Dense Passage Retrieval
Pre-trained language model (PTM) has been shown to yield powerful text representations for dense passage retrieval task. The Masked Language Modeling (MLM) is a major sub-task of the pre-training process. However, we fou…
Language ModelingLanguage ModellingMasked Language ModelingPassage Retrieval+1Towards Unifying Reference Expression Generation and Comprehension
Reference Expression Generation (REG) and Comprehension (REC) are two highly correlated tasks. Modeling REG and REC simultaneously for utilizing the relation between them is a promising way to improve both. However, the …
Language ModelingLanguage ModellingMasked Language ModelingReference Expression GenerationGenerative Prompt Tuning for Relation Classification
Using prompts to explore the knowledge contained within pre-trained language models for downstream tasks has now become an active topic. Current prompt tuning methods mostly convert the downstream tasks to masked languag…
ClassificationLanguage ModelingLanguage ModellingMasked Language Modeling+4InforMask: Unsupervised Informative Masking for Language Model Pretraining
Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this pape…
Language ModelingLanguage ModellingMasked Language Modelingmodel+2SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
Named geographic entities (geo-entities for short) are the building blocks of many geographic datasets. Characterizing geo-entities is integral to various application domains, such as geo-intelligence and map comprehensi…
Entity LinkingEntity TypingLanguage ModelingLanguage Modelling+1Deep Bidirectional Language-Knowledge Graph Pretraining
Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provide…
Common Sense ReasoningKnowledge GraphsLanguage ModelingLanguage Modelling+4