Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese
Despite their successes in NLP, Transformer-based language models still require extensive computing resources and suffer in low-resource or low-compute settings. In this paper, we present AxomiyaBERTa, a novel BERT model…
Language ModelingLanguage ModellingMasked Language Modelingmodel+3Federated Learning of Medical Concepts Embedding using BEHRT
Electronic Health Records (EHR) data contains medical records such as diagnoses, medications, procedures, and treatments of patients. This data is often considered sensitive medical information. Therefore, the EHR data f…
Federated LearningLanguage ModelingLanguage ModellingMasked Language ModelingExtrapolating Multilingual Understanding Models as Multilingual Generators
Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these non-autoregressive (NAR) …
DenoisingLanguage ModelingLanguage ModellingMachine Translation+5Bidirectional Transformer Reranker for Grammatical Error Correction
Pre-trained seq2seq models have achieved state-of-the-art results in the grammatical error correction task. However, these models still suffer from a prediction bias due to their unidirectional decoding. Thus, we propose…
DecoderGrammatical Error CorrectionLanguage ModelingLanguage Modelling+3A Pilot Study on Dialogue-Level Dependency Parsing for Chinese
Dialogue-level dependency parsing has received insufficient attention, especially for Chinese. To this end, we draw on ideas from syntactic dependency and rhetorical structure theory (RST), developing a high-quality huma…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language ModelingPatton: Language Model Pretraining on Text-Rich Networks
A real-world text corpus sometimes comprises not only text documents but also semantic links between them (e.g., academic papers in a bibliographic network are linked by citations and co-authorships). Text documents and …
Language ModelingLanguage ModellingMasked Language Modelingmodel+1ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain
The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill c…
De-identificationLanguage ModelingLanguage ModellingMasked Language Modeling+1Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model
In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective. We demonstrate that a nearly identical model architect…
Language ModelingLanguage ModellingMasked Language ModelingSegmentation+2How does the task complexity of masked pretraining objectives affect downstream performance?
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally ef…
Language ModelingLanguage ModellingMasked Language ModelingDinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show t…
ClusteringLanguage ModelingLanguage ModellingMasked Language Modeling+3Pre-training Language Model as a Multi-perspective Course Learner
ELECTRA, the generator-discriminator pre-training framework, has achieved impressive semantic construction capability among various downstream tasks. Despite the convincing performance, ELECTRA still faces the challenges…
Language ModelingLanguage ModellingMasked Language ModelingMapping of attention mechanisms to a generalized Potts model
Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked la…
Language ModelingLanguage ModellingMasked Language ModelingUnsupervised Improvement of Factual Knowledge in Language Models
Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we…
Language ModelingLanguage ModellingMasked Language ModelingNatural Language Inference+2PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation
Multilingual pre-training significantly improves many multilingual NLP tasks, including machine translation. Most existing methods are based on some variants of masked language modeling and text-denoising objectives on m…
DenoisingLanguage ModelingLanguage ModellingMachine Translation+5Joint unsupervised and supervised learning for context-aware language identification
Language identification (LID) recognizes the language of a spoken utterance automatically. According to recent studies, LID models trained with an automatic speech recognition (ASR) task perform better than those trained…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language IdentificationLanguage Modeling+4Fine-grained Audible Video Description
We explore a new task for audio-visual-language modeling called fine-grained audible video description (FAVD). It aims to provide detailed textual descriptions for the given audible videos, including the appearance and s…
Language ModelingLanguage ModellingMasked Language ModelingSentence+4Accelerating Vision-Language Pretraining with Free Language Modeling
The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training time, especially on large-scale web dataset…
GPULanguage ModelingLanguage ModellingMasked Language Modeling+1Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Pri…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+9HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation
Recent works attempt to employ pre-training in Vision-and-Language Navigation (VLN). However, these methods neglect the importance of historical contexts or ignore predicting future actions during pre-training, limiting …
Decision MakingLanguage ModelingLanguage ModellingMasked Language Modeling+1CCPL: Cross-modal Contrastive Protein Learning
Effective protein representation learning is crucial for predicting protein functions. Traditional methods often pretrain protein language models on large, unlabeled amino acid sequences, followed by finetuning on labele…
Language ModelingLanguage ModellingMasked Language ModelingProtein Design+2