Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
Vision-language pre-training (VLP) methods are blossoming recently, and its crucial goal is to jointly learn visual and textual features via a transformer-based architecture, demonstrating promising improvements on a var…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+1Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
In this paper, we for the first time explore helpful multi-modal contextual knowledge to understand novel categories for open-vocabulary object detection (OVD). The multi-modal contextual knowledge stands for the joint r…
Knowledge DistillationLanguage ModelingLanguage ModellingMasked Language Modeling+5A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER
The objective of few-shot named entity recognition is to identify named entities with limited labeled instances. Previous works have primarily focused on optimizing the traditional token-wise classification framework, wh…
Contrastive Learningfew-shot-nerFew-shot NERLanguage Modeling+5Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
The recent surge of generative AI has been fueled by the generative power of diffusion probabilistic models and the scalable capabilities of large language models. Despite their potential, it remains elusive whether diff…
In-Context LearningLanguage ModelingLanguage ModellingMasked Language ModelingPre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval
Grounded on pre-trained language models (PLMs), dense retrieval has been studied extensively on plain text. In contrast, there has been little research on retrieving data with multiple aspects using dense models. In the …
Language ModelingLanguage ModellingMasked Language ModelingRetrievalLatent State Models of Training Dynamics
The impact of randomness on model training is poorly understood. How do differences in data order and initialization actually manifest in the model, such that some training runs outperform others or converge faster? Furt…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
In this paper, we investigate the in-context learning ability of retrieval-augmented encoder-decoder language models. We first conduct a comprehensive analysis of existing models and identify their limitations in in-cont…
DecoderIn-Context LearningLanguage ModelingLanguage Modelling+3Pairing interacting protein sequences using masked language modeling
Predicting which proteins interact together from amino-acid sequences is an important task. We develop a method to pair interacting protein sequences which leverages the power of protein language models trained on multip…
Language ModelingLanguage ModellingMasked Language ModelingProtein Language ModelStochastic positional embeddings improve masked image modeling
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images. Despite its recent success, learning good representations through MIM remains challenging because …
Language ModellingMasked Language ModelingSelf-Supervised LearningVideo Segmentation+1GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
Large-scale language models such as DNABert and LOGO aim to learn optimal gene representations and are trained on the entire Human Reference Genome. However, standard tokenization schemes involve a simple sliding window …
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language ModelingPASTA: Pretrained Action-State Transformer Agents
Self-supervised learning has brought about a revolutionary paradigm shift in various computing domains, including NLP, vision, and biology. Recent approaches involve pre-training transformer models on vast amounts of unl…
Language ModelingLanguage ModellingMasked Language ModelingOffline RL+2Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models
Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and …
Language ModelingLanguage ModellingMasked Language ModelingImproving the Reusability of Pre-trained Language Models in Real-world Applications
The reusability of state-of-the-art Pre-trained Language Models (PLMs) is often limited by their generalization problem, where their performance drastically decreases when evaluated on examples that differ from the train…
Language ModelingLanguage ModellingMasked Language ModelingImproving BERT with Hybrid Pooling Network and Drop Mask
Transformer-based pre-trained language models, such as BERT, achieve great success in various natural language understanding tasks. Prior research found that BERT captures a rich hierarchy of linguistic information at di…
Language ModelingLanguage ModellingMasked Language ModelingNatural Language Understanding+1Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling
In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where the objective is altered from the reconstruction of the exact identity of randomly selected masked subwo…
Language ModelingLanguage ModellingMasked Language ModelingBiomedical Language Models are Robust to Sub-optimal Tokenization
As opposed to general English, many concepts in biomedical terminology have been designed in recent history by biomedical professionals with the goal of being precise and concise. This is often achieved by concatenating …
Entity LinkingLanguage ModelingLanguage ModellingMasked Language Modeling+4S2SNet: A Pretrained Neural Network for Superconductivity Discovery
Superconductivity allows electrical current to flow without any energy loss, and thus making solids superconducting is a grand goal of physics, material science, and electrical engineering. More than 16 Nobel Laureates h…
Electrical EngineeringLanguage ModelingLanguage ModellingMasked Language ModelingSolving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling
Enhancing AI systems with efficient communication skills that align with human understanding is crucial for their effective assistance to human users. Proactive initiatives from the system side are needed to discern spec…
Language ModelingLanguage ModellingMasked Language ModelingMinecraftInvestigating Masking-based Data Generation in Language Models
The current era of natural language processing (NLP) has been defined by the prominence of pre-trained language models since the advent of BERT. A feature of BERT and models with similar architecture is the objective of …
Data AugmentationLanguage ModelingLanguage ModellingMasked Language ModelingPersonalized Image Enhancement Featuring Masked Style Modeling
We address personalized image enhancement in this study, where we enhance input images for each user based on the user's preferred images. Previous methods apply the same preferred style to all input images (i.e., only o…
Image EnhancementLanguage ModelingLanguage ModellingMasked Language Modeling