Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification
Recently, prompt-based fine-tuning has garnered considerable interest as a core technique for few-shot text classification task. This approach reformulates the fine-tuning objective to align with the Masked Language Mode…
ClassificationFew-Shot Text ClassificationLanguage ModelingLanguage Modelling+4LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models
Vision-language pre-training like CLIP has shown promising performance on various downstream tasks such as zero-shot image classification and image-text retrieval. Most of the existing CLIP-alike works usually adopt rela…
image-classificationImage ClassificationImage-text RetrievalLanguage Modeling+4BIM: Block-Wise Self-Supervised Learning with Masked Image Modeling
Like masked language modeling (MLM) in natural language processing, masked image modeling (MIM) aims to extract valuable insights from image patches to enhance the feature extraction capabilities of the underlying deep n…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+1CLIMB: Curriculum Learning for Infant-inspired Model Building
We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge. The challenge requires training a language model from scratch using only a relatively small training dataset of ten million words. We…
Language ModelingLanguage ModellingMasked Language ModelingmodelUser Persona Identification and New Service Adaptation Recommendation
Providing a personalized user experience on information dense webpages helps users in reaching their end-goals sooner. We explore an automated approach to identifying user personas by leveraging high dimensional trajecto…
Collaborative FilteringLanguage ModelingLanguage ModellingMasked Language ModelingUnderstanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision
This paper presents the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED) foundation model, analyzing DNA sequences at byte-level precision with an encoder-decoder Transformer architecture. ENBED uses a sub-quadrati…
DecoderLanguage ModelingLanguage ModellingMasked Language ModelingBERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text
Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT'…
Language ModelingLanguage ModellingMasked Language ModelingCounterfactually Probing Language Identity in Multilingual Models
Techniques in causal analysis of language models illuminate how linguistic information is organized in LLMs. We use one such technique, AlterRep, a method of counterfactual probing, to explore the internal structure of m…
counterfactualLanguage ModelingLanguage ModellingMasked Language Modeling+1Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
We present Lil-Bevo, our submission to the BabyLM Challenge. We pretrained our masked language models with three ingredients: an initial pretraining with music data, training on shorter sequences before training on longe…
Language ModelingLanguage ModellingMasked Language ModelingThe Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
We analyze the masked language modeling pretraining objective function from the perspective of the distributional hypothesis. We investigate whether better sample efficiency and the better generalization capability of mo…
Language ModelingLanguage ModellingMasked Language ModelingSemantic Similarity+1GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
Humans subconsciously engage in geospatial reasoning when reading articles. We recognize place names and their spatial relations in text and mentally associate them with their physical locations on Earth. Although pretra…
ArticlesContrastive LearningEntity TypingLanguage Modeling+4DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias
Numerous debiasing techniques have been proposed to mitigate the gender bias that is prevalent in pretrained language models. These are often evaluated on datasets that check the extent to which the model is gender-neutr…
Language ModelingLanguage ModellingMasked Language ModelingFATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
Sequential tabular data is one of the most commonly used data types in real-world applications. Different from conventional tabular data, where rows in a table are independent, sequential tabular data contains rich conte…
Language ModelingLanguage ModellingMasked Language ModelingFiLM: Fill-in Language Models for Any-Order Generation
Language models have become the backbone of today's AI systems. However, their predominant left-to-right generation limits the use of bidirectional context, which is essential for tasks that involve filling text in the m…
Language ModelingLanguage ModellingLarge Language ModelMasked Language ModelingClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction
Click-through rate (CTR) prediction has become increasingly indispensable for various Internet applications. Traditional CTR models convert the multi-field categorical data into ID features via one-hot encoding, and extr…
Click-Through Rate PredictionLanguage ModelingLanguage ModellingMasked Language ModelingEnhancing BERT-Based Visual Question Answering through Keyword-Driven Sentence Selection
The Document-based Visual Question Answering competition addresses the automatic detection of parent-child relationships between elements in multi-page documents. The goal is to identify the document elements that answer…
Language ModelingLanguage ModellingMasked Language ModelingQuestion Answering+2PepMLM: Target Sequence-Conditioned Generation of Therapeutic Peptide Binders via Span Masked Language Modeling
Target proteins that lack accessible binding pockets and conformational stability have posed increasing challenges for drug development. Induced proximity strategies, such as PROTACs and molecular glues, have thus gained…
BenchmarkingLanguage ModelingLanguage ModellingMasked Language ModelingStructural Self-Supervised Objectives for Transformers
This thesis focuses on improving the pre-training of natural language models using unsupervised raw data to make them more efficient and aligned with downstream applications. In the first part, we introduce three alterna…
Fact VerificationLanguage ModelingLanguage ModellingMasked Language Modeling+2PerPLM: Personalized Fine-tuning of Pretrained Language Models via Writer-specific Intermediate Learning and Prompts
The meanings of words and phrases depend not only on where they are used (contexts) but also on who use them (writers). Pretrained language models (PLMs) are powerful tools for capturing context, but they are typically p…
Language ModelingLanguage ModellingMasked Language ModelingBiLMa: Bidirectional Local-Matching for Text-based Person Re-identification
Text-based person re-identification (TBPReID) aims to retrieve person images represented by a given textual query. In this task, how to effectively align images and texts globally and locally is a crucial challenge. Rece…
Image to textLanguage ModelingLanguage ModellingMasked Language Modeling+1