Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
TemPrompt: Multi-Task Prompt Learning for Temporal Relation Extraction in RAG-based Crowdsourcing Systems
Temporal relation extraction (TRE) aims to grasp the evolution of events or actions, and thus shape the workflow of associated tasks, so it holds promise in helping understand task requests initiated by requesters in cro…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+5QueerBench: Quantifying Discrimination in Language Models Toward Queer Identities
With the increasing role of Natural Language Processing (NLP) in various applications, challenges concerning bias and stereotype perpetuation are accentuated, which often leads to hate speech and harm. Despite existing s…
Language ModelingLanguage ModellingMasked Language ModelingSentenceSimple and Effective Masked Diffusion Language Models
While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple m…
Language ModelingLanguage ModellingMasked Language ModelingVCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images. This task stems from the observatio…
Language ModelingLanguage ModellingMasked Language ModelingOptical Character Recognition+2Seventeenth-Century Spanish American Notary Records for Fine-Tuning Spanish Large Language Models
Large language models have gained tremendous popularity in domains such as e-commerce, finance, healthcare, and education. Fine-tuning is a common approach to customize an LLM on a domain-specific dataset for a desired d…
Language ModelingLanguage ModellingMasked Language ModelingTowards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models
In the evolving field of Natural Language Processing, understanding the temporal context of text is increasingly crucial. This study investigates methods to incorporate temporal information during pre-training, aiming to…
Document DatingLanguage ModelingLanguage ModellingMasked Language ModelingMasked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis
In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditiona…
Density EstimationImputationLanguage ModelingLanguage Modelling+4Knowledge-enhanced Prompt Tuning for Dialogue-based Relation Extraction with Trigger and Label Semantic
Dialogue-based relation extraction (DRE) aims to determine the semantic relation of a given pair of arguments from a piece of dialogue, which has received increasing attention. Due to the low information density of dia…
Language ModelingLanguage ModellingMasked Language ModelingRelation+2Transformer based neural networks for emotion recognition in conversations
This paper outlines the approach of the ISDS-NLP team in the SemEval 2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF). For Subtask 1 we obtained a weighted F1 score of 0.43 and placed 12 in…
Causal Language ModelingEmotion ClassificationEmotion RecognitionLanguage Modeling+3Self-Distillation Improves DNA Sequence Inference
Self-supervised pretraining (SSP) has been recognized as a method to enhance prediction accuracy in various downstream tasks. However, its efficacy for DNA sequences remains somewhat constrained. This limitation stems pr…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language ModelingKnowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget
Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target stu…
Knowledge DistillationLanguage ModelingLanguage ModellingMasked Language ModelingPromptCL: Improving Event Representation via Prompt Template and Contrastive Learning
The representation of events in text plays a significant role in various NLP tasks. Recent research demonstrates that contrastive learning has the ability to improve event comprehension capabilities of Pre-trained Langua…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Emerging Property of Masked Token for Effective Pre-training
Driven by the success of Masked Language Modeling (MLM), the realm of self-supervised learning for computer vision has been invigorated by the central role of Masked Image Modeling (MIM) in driving recent breakthroughs. …
AttributeLanguage ModelingLanguage ModellingMasked Language Modeling+1OPSD: an Offensive Persian Social media Dataset and its baseline evaluations
The proliferation of hate speech and offensive comments on social media has become increasingly prevalent due to user activities. Such comments can have detrimental effects on individuals' psychological well-being and so…
Language ModelingLanguage ModellingMasked Language ModelingSyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
Vision-language models (VLMs) have made significant strides in cross-modal understanding through large-scale paired datasets. However, in fashion domain, datasets often exhibit a disparity between the information conveye…
Contrastive LearningImage-text matchingLanguage ModelingLanguage Modelling+2Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Tickets
Current soft prompt methods yield limited performance when applied to small-sized models (fewer than a billion parameters). Deep prompt-tuning, which entails prepending parameters in each layer for enhanced efficacy, pre…
Language ModelingLanguage ModellingMasked Language ModelingPrompt LearningDeveloping Healthcare Language Model Embedding Spaces
Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. We explore specialized pre-training to adapt smaller LLMs to different healthcare datasets. Three methods ar…
Contrastive LearningDocument ClassificationLanguage ModelingLanguage Modelling+2Fingerprinting web servers through Transformer-encoded HTTP response headers
We explored leveraging state-of-the-art deep learning, big data, and natural language processing to enhance the detection of vulnerable web server versions. Focusing on improving accuracy and specificity over rule-based …
Language ModelingLanguage ModellingMasked Language ModelingSpecificityDetecting Bias in Large Language Models: Fine-tuned KcBERT
The rapid advancement of large language models (LLMs) has enabled natural language processing capabilities similar to those of humans, and LLMs are being widely utilized across various societal domains such as education …
Language ModelingLanguage ModellingMasked Language ModelingMerging Text Transformer Models from Different Initializations
Recent work on permutation-based model merging has shown impressive low- or zero-barrier mode connectivity between models from completely different initializations. However, this line of work has not yet extended to the …
Language ModelingLanguage ModellingMasked Language Modeling