Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation
The excellent generalization ability of self-supervised learning (SSL) for speech foundation models has garnered significant attention. HuBERT is a successful example that utilizes offline clustering to convert speech fe…
Automatic Speech RecognitionClusteringLanguage ModelingLanguage Modelling+3Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models
Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency …
DecoderLanguage ModelingLanguage ModellingMasked Language ModelingGenerate to Understand for Representation
In recent years, a significant number of high-quality pretrained models have emerged, greatly impacting Natural Language Understanding (NLU), Natural Language Generation (NLG), and Text Representation tasks. Traditionall…
Contrastive LearningGPULanguage ModelingLanguage Modelling+3Global and Local Semantic Completion Learning for Vision-Language Pre-training
Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, numerous masked modeling tasks have…
cross-modal alignmentImage-text RetrievalLanguage ModellingMasked Language Modeling+5Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems
Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored…
Conversational Response SelectionDecoderLanguage ModelingLanguage Modelling+2Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization
Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. The generated summary can save the time of reading many documents by providing the important con…
Abstractive Text SummarizationDecoderDocument SummarizationLanguage Modeling+4On the Difference of BERT-style and CLIP-style Text Encoders
Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) ha…
Image GenerationLanguage ModelingLanguage ModellingMasked Language Modeling+2Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction
Task-oriented dialogues often require agents to enact complex, multi-step procedures in order to meet user requests. While large language models have found success automating these dialogues in constrained environments, …
Language ModelingLanguage ModellingLarge Language ModelMasked Language Modeling+3Fair multilingual vandalism detection system for Wikipedia
This paper presents a novel design of the system aimed at supporting the Wikipedia community in addressing vandalism on the platform. To achieve this, we collected a massive dataset of 47 languages, and applied advanced …
Feature EngineeringLanguage ModelingLanguage ModellingMasked Language ModelingUnderstanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression
Data augmentation is critical to the empirical success of modern self-supervised representation learning, such as contrastive learning and masked language modeling. However, a theoretical understanding of the exact role …
Contrastive LearningData AugmentationGeneralization BoundsLanguage Modeling+6LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding
Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years. Pre-trained models on a large number of document images with transformer-based backbones have led to significant perf…
document-image-classificationDocument Image Classificationdocument understandingimage-classification+9Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models
Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through fine-tuning. However, fine-tuned models still suffer from overconfident predictions, esp…
Language ModelingLanguage ModellingMasked Language ModelingNatural Language UnderstandingAdapting Learned Sparse Retrieval for Long Documents
Learned sparse retrieval (LSR) is a family of neural retrieval methods that transform queries and documents into sparse weight vectors aligned with a vocabulary. While LSR approaches like Splade work well for short passa…
Language ModelingLanguage ModellingMasked Language ModelingRetrievalRethinking Masked Language Modeling for Chinese Spelling Correction
In this paper, we study Chinese Spelling Correction (CSC) as a joint decision made by two separate models: a language model and an error model. Through empirical analysis, we find that fine-tuning BERT tends to over-fit …
DiversityDomain GeneralizationLanguage ModelingLanguage Modelling+2An Investigation of Noise in Morphological Inflection
With a growing focus on morphological inflection systems for languages where high-quality data is scarce, training data noise is a serious but so far largely ignored concern. We aim at closing this gap by investigating t…
Language ModelingLanguage ModellingMasked Language ModelingMorphological InflectionHoney, I Shrunk the Language: Language Model Behavior at Reduced Scale
In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter…
Language ModelingLanguage ModellingMasked Language ModelingMasked and Permuted Implicit Context Learning for Scene Text Recognition
Scene Text Recognition (STR) is difficult because of the variations in text styles, shapes, and backgrounds. Though the integration of linguistic information enhances models' performance, existing methods based on either…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+1Dynamic Masking Rate Schedules for MLM Pretraining
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout trainin…
Language ModelingLanguage ModellingMasked Language ModelingSchedulingSelf-Evolution Learning for Discriminative Language Model Pretraining
Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not consider the importance of the different words …
Language ModelingLanguage ModellingMasked Language Modelingmodel+1Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection
Event detection is a crucial information extraction task in many domains, such as Wikipedia or news. The task typically relies on trigger detection (TD) -- identifying token spans in the text that evoke specific events. …
Event DetectionLanguage ModelingLanguage ModellingMasked Language Modeling+1