Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Foundation Posteriors for Approximate Probabilistic Inference
Probabilistic programs provide an expressive representation language for generative models. Given a probabilistic program, we are interested in the task of posterior inference: estimating a latent variable given a set of…
Language ModelingLanguage ModellingMasked Language ModelingVariational InferenceAn Empirical Study Of Self-supervised Learning Approaches For Object Detection With Transformers
Self-supervised learning (SSL) methods such as masked language modeling have shown massive performance gains by pretraining transformer models for a variety of natural language processing tasks. The follow-up research ad…
image-classificationImage ClassificationImage ReconstructionLanguage Modeling+6Towards Unified Prompt Tuning for Few-shot Text Classification
Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot text classification by employing task-specific prompts. Yet, PLMs are unfamiliar with prompt-style expressions during…
ClassificationFew-Shot LearningFew-Shot Text ClassificationLanguage Modeling+6KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering
Extractive Question Answering (EQA) is one of the most important tasks in Machine Reading Comprehension (MRC), which can be solved by fine-tuning the span selecting heads of Pre-trained Language Models (PLMs). However, m…
Contrastive LearningExtractive Question-AnsweringFew-Shot LearningLanguage Modeling+5Declaration-based Prompt Tuning for Visual Question Answering
In recent years, the pre-training-then-fine-tuning paradigm has yielded immense success on a wide spectrum of cross-modal tasks, such as visual question answering (VQA), in which a visual-language (VL) model is first opt…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+5Contrastive Learning for Prompt-Based Few-Shot Language Learners
The impressive performance of GPT-3 using natural language prompts and in-context learning has inspired work on better fine-tuning of moderately-sized models under this paradigm. Following this line of work, we present a…
Contrastive LearningIn-Context LearningLanguage ModelingLanguage Modelling+1“Is Whole Word Masking Always Better for Chinese BERT?”: Probing on Chinese Grammatical Error Correction
Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The…
Grammatical Error CorrectionLanguage ModelingLanguage ModellingMasked Language Modeling+1DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog
Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD…
dialog state trackingLanguage ModelingLanguage ModellingMasked Language Modeling+1Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates
Cross-lingual natural language inference (XNLI) is a fundamental task in cross-lingual natural language understanding. Recently this task is commonly addressed by pre-trained cross-lingual language models. Existing metho…
Cross-Lingual Natural Language InferenceCross-Lingual TransferLanguage ModelingLanguage Modelling+4Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis
Cross-domain sentiment analysis has achieved promising results with the help of pre-trained language models. As GPT-3 appears, prompt tuning has been widely explored to enable better semantic modeling in many natural lan…
Domain AdaptationLanguage ModelingLanguage ModellingMasked Language Modeling+1Unsupervised Dependency Graph Network
Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, some self-attention heads correspond well to individual dependency types. Inspire…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language Modeling+3Phrase-aware Unsupervised Constituency Parsing
Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. Despite their high accuracy in identifying low-level structures, prior arts tend to…
Constituency ParsingLanguage ModelingLanguage ModellingMasked Language ModelingVision-Language Pre-Training for Boosting Scene Text Detectors
Recently, vision-language joint representation learning has proven to be highly effective in various scenarios. In this paper, we specifically adapt vision-language joint learning for scene text detection, a task that in…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+3A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer
Being a popular mode of text-based communication in multilingual communities, code-mixing in online social media has became an important subject to study. Learning the semantics and morphology of code-mixed language rema…
Language ModelingLanguage ModellingMasked Language ModelingRepresentation Learning+2Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors
Chinese BERT models achieve remarkable progress in dealing with grammatical errors of word substitution. However, they fail to handle word insertion and deletion because BERT assumes the existence of a word at each posit…
Language ModelingLanguage ModellingMasked Language ModelingPositionUnsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking
Players of online games generate rich behavioral data during gaming. Based on these data, game developers can build a range of data science applications, such as bot detection and social recommendation, to improve the ga…
Feature EngineeringLanguage ModelingLanguage ModellingMasked Language Modeling+1LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text mod…
cross-modal alignmentDocument AIdocument-image-classificationDocument Image Classification+16SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words
Pre-trained models are widely used in the tasks of natural language processing nowadays. However, in the specific field of text simplification, the research on improving pre-trained models is still blank. In this work, w…
Language ModelingLanguage ModellingLexical SimplificationMasked Language Modeling+2WordAlchemy: A transformer-based Reverse Dictionary
A reverse dictionary takes a target word's description as input and returns the words that fit the description. Reverse Dictionaries are useful for new language learners, anomia patients, and for solving common tip-of-th…
Language ModelingLanguage ModellingMasked Language ModelingReverse Dictionary+1Text Revision by On-the-Fly Representation Optimization
Text revision refers to a family of natural language generation tasks, where the source and target sequences share moderate resemblance in surface form but differentiate in attributes, such as text formality and simplici…
AttributeLanguage ModelingLanguage ModellingMasked Language Modeling+2