Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
Large-scale vision-language pre-trained (VLP) models are prone to hallucinate non-existent visual objects when generating text based on visual information. In this paper, we systematically study the object hallucination …
HallucinationImage AugmentationLanguage ModellingMasked Language Modeling+2MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intr…
Contrastive LearningImage-text matchingImage-text RetrievalLanguage Modeling+10Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training
Recently, knowledge-enhanced pre-trained language models (KEPLMs) improve context-aware representations via learning from structured relations in knowledge graphs, and/or linguistic knowledge from syntactic or dependency…
GPUKnowledge GraphsLanguage ModelingLanguage Modelling+2Mixture of Attention Heads: Selecting Attention Heads Per Token
Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing. However, the study of MoE components mostly focused on the feedforward layer in Tra…
Computational EfficiencyLanguage ModelingLanguage ModellingMachine Translation+2The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection
This paper studies the problem of injecting factual knowledge into large pre-trained language models. We train adapter modules on parts of the ConceptNet knowledge graph using the masked language modeling objective and e…
Language ModelingLanguage ModellingMasked Language ModelingTowards Making the Most of Pre-trained Translation Model for Quality Estimation
“Machine translation quality estimation (QE) aims to evaluate the quality of machine translation automatically without relying on any reference. One common practice is applying the translation model as a feature extracto…
DenoisingLanguage ModelingLanguage ModellingMachine Translation+3Taking Actions Separately: A Bidirectionally-Adaptive Transfer Learning Method for Low-Resource Neural Machine Translation
Training Neural Machine Translation (NMT) models suffers from sparse parallel data, in the infrequent translation scenarios towards low-resource source languages. The existing solutions primarily concentrate on the utili…
Generative Adversarial NetworkLanguage ModelingLanguage ModellingLow Resource Neural Machine Translation+8A Closer Look at Parameter Contributions When Training Neural Language and Translation Models
We analyze the learning dynamics of neural language and translation models using Loss Change Allocation (LCA), an indicator that enables a fine-grained analysis of parameter updates when optimizing for the loss function.…
Causal Language ModelingLanguage ModelingLanguage ModellingMachine Translation+2KUL@SMM4H’22: Template Augmented Adaptive Pre-training for Tweet Classification
This paper describes models developed for the Social Media Mining for Health (SMM4H) 2022 shared tasks. Our team participated in the first subtask that classifies tweets with Adverse Drug Effect (ADE) mentions. Our best-…
Data AugmentationLanguage ModelingLanguage ModellingMasked Language ModelingBidirectional Language Models Are Also Few-shot Learners
Large language models such as GPT-3 (Brown et al., 2020) can perform arbitrary tasks without undergoing fine-tuning after being prompted with only a few labeled examples. An arbitrary task can be reformulated as a natura…
DenoisingLanguage ModelingLanguage ModellingMachine Translation+3IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
In this paper, we describe our participation in the subtask 1 of CASE-2022, Event Causality Identification with Casual News Corpus. We address the Causal Relation Identification (CRI) task by exploiting a set of simple y…
Event Causality IdentificationLanguage ModelingLanguage ModellingMasked Language ModelingTransPolymer: a Transformer-based language model for polymer property predictions
Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Rece…
Language ModelingLanguage ModellingMasked Language ModelingPrediction+2Learning Better Masking for Better Language Model Pre-training
Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied…
DenoisingLanguage ModelingLanguage ModellingMasked Language ModelingImage as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both…
AllCross-Modal RetrievalImage Captioningimage-classification+13GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training
Most of the currently existing vision and language pre-training (VLP) methods have mainly focused on how to extract and align vision and text features. In contrast to the mainstream VLP methods, we highlight that two rou…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+1Towards No.1 in CLUE Semantic Matching Challenge: Pre-trained Language Model Erlangshen with Propensity-Corrected Loss
This report describes a pre-trained language model Erlangshen with propensity-corrected loss, the No.1 in CLUE Semantic Matching Challenge. In the pre-training stage, we construct a dynamic masking strategy based on know…
Language ModelingLanguage ModellingMasked Language ModelingMasked Vision and Language Modeling for Multi-modal Representation Learning
In this paper, we study how to use masked signal modeling in vision and language (V+L) representation learning. Instead of developing masked language modeling (MLM) and masked image modeling (MIM) independently, we propo…
cross-modal alignmentLanguage ModelingLanguage ModellingMasked Language Modeling+1Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
Language modality within the vision language pretraining framework is innately discretized, endowing each word in the language vocabulary a semantic meaning. In contrast, visual modality is inherently continuous and high…
Language ModelingLanguage ModellingMasked Language ModelingBoosting Point-BERT by Multi-choice Tokens
Masked language modeling (MLM) has become one of the most successful self-supervised pre-training task. Inspired by its success, Point-BERT, as a pioneer work in point cloud, proposed masked point modeling (MPM) to pre-t…
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Unsupervised pre-training of graph transformers on patient population graphs
Pre-training has shown success in different areas of machine learning, such as Computer Vision, Natural Language Processing (NLP), and medical imaging. However, it has not been fully explored for clinical data analysis. …
Language ModelingLanguage ModellingMasked Language ModelingTransfer Learning+1