Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Misinformation Detection in Social Media Video Posts
With the growing adoption of short-form video by social media platforms, reducing the spread of misinformation through video posts has become a critical challenge for social media providers. In this paper, we develop met…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Prompt-Guided Injection of Conformation to Pre-trained Protein Model
Pre-trained protein models (PTPMs) represent a protein with one fixed embedding and thus are not capable for diverse tasks. For example, protein structures can shift, namely protein folding, between several conformations…
Language ModelingLanguage ModellingMasked Language ModelingProtein FoldingSelf-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to assoc…
Language ModelingLanguage ModellingMasked Language ModelingRepresentation Learning+1MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
Previous vision-language pre-training models mainly construct multi-modal inputs with tokens and objects (pixels) followed by performing cross-modality interaction between them. We argue that the input of only tokens and…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+2Text Style Transfer for Bias Mitigation using Masked Language Modeling
It is well known that textual data on the internet and other digital platforms contain significant levels of bias and stereotypes. Although many such texts contain stereotypes and biases that inherently exist in natural …
Language ModelingLanguage ModellingMasked Language ModelingStyle Transfer+1STT: Soft Template Tuning for Few-Shot Learning
With the rapid expansion of large pre-trained language models, fine-tuning all the model parameters for downstream tasks is becoming computationally prohibitive. The recently developed prompt-based methods freeze the ent…
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Causal Distillation for Language Models
Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. The standard approach to distillation trains a student model against two objectives: a task-speci…
Language ModelingLanguage ModellingMasked Language Modelingnamed-entity-recognition+4Data Augmentation for Biomedical Factoid Question Answering
We study the effect of seven data augmentation (DA) methods in factoid question answering, focusing on the biomedical domain, where obtaining training instances is particularly difficult. We experiment with data from the…
Data AugmentationInformation RetrievalLanguage ModelingLanguage Modelling+8Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizer
Recent advances on prompt-tuning cast few-shot classification tasks as a masked language modeling problem. By wrapping input into a template and using a verbalizer which constructs a mapping between label space and label…
ClassificationContrastive LearningLanguage ModelingLanguage Modelling+3A Transfer Learning Pipeline for Educational Resource Discovery with Application in Leading Paragraph Generation
Effective human learning depends on a wide selection of educational materials that align with the learner's current understanding of the topic. While the Internet has revolutionized human learning or education, a substan…
Information RetrievalLanguage ModelingLanguage ModellingMasked Language Modeling+3EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
Mixture-of-experts (MoE) is becoming popular due to its success in improving the model quality, especially in Transformers. By routing tokens with a sparse gate to a few experts (i.e., a small pieces of the full model), …
Language ModelingLanguage ModellingMachine TranslationMasked Language Modeling+1Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge
Transformer models pre-trained with a masked-language-modeling objective (e.g., BERT) encode commonsense knowledge as evidenced by behavioral probes; however, the extent to which this knowledge is acquired by systematic …
Language ModelingLanguage ModellingMasked Language ModelingOpen-Ended Question AnsweringCoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
BERT-type structure has led to the revolution of vision-language pre-training and the achievement of state-of-the-art results on numerous vision-language downstream tasks. Existing solutions dominantly capitalize on the …
Cross-Modal RetrievalDecoderDenoisingLanguage Modeling+6Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like objectives (masked language modeling and image-text matching) during pretraining. Although they perform well in many under…
Image-text matchingImage-text RetrievalLanguage ModelingLanguage Modelling+9Causal Distillation for Language Models
Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. The standard approach to distillation trains a student model against two objectives: a task-speci…
Language ModelingLanguage ModellingMasked Language Modelingnamed-entity-recognition+4DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
Prior research in the area of Natural Language Processing (NLP) has shown that including the syntactic structure of a sentence using a dependency parse tree while training a representation learning model improves the per…
Language ModelingLanguage ModellingMasked Language ModelingNatural Language Inference+5UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e.g., image or language) or multimodal inputs (e.g., the concatenation of the image and the question), …
Image CaptioningImage-text matchingImage-text RetrievalLanguage Modeling+10LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model
The system log generated in a computer system refers to large-scale data that are collected simultaneously and used as the basic data for determining errors, intrusion and abnormal behaviors. The aim of system log anomal…
Anomaly DetectionLanguage ModelingLanguage ModellingMasked Language ModelingDAWSON: Data Augmentation using Weak Supervision On Natural Language
We propose a novel data augmentation model for text, using all available data through weak supervision. To improve generalization, recent work in the field uses BERT and masked language modeling to conditionally augment …
Data AugmentationLanguage ModelingLanguage ModellingMasked Language Modeling+4Unsupervised Dependency Graph Network
Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, some self-attention heads correspond well to individual dependency types. Inspire…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language Modeling+3