Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents
While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechan…
Document ClassificationLanguage ModelingLanguage ModellingMasked Language Modeling+3Substrate Prediction for RiPP Biosynthetic Enzymes via Masked Language Modeling and Transfer Learning
Ribosomally synthesized and post-translationally modified peptide (RiPP) biosynthetic enzymes often exhibit promiscuous substrate preferences that cannot be reduced to simple rules. Large language models are promising to…
Language ModelingLanguage ModellingMasked Language ModelingTransfer LearningMeasuring Social Biases in Masked Language Models by Proxy of Prediction Quality
Innovative transformer-based language models produce contextually-aware token embeddings and have achieved state-of-the-art performance for a variety of natural language tasks, but have been shown to encode unwanted bias…
Language ModelingLanguage ModellingMasked Language ModelingCodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking
Transformer based code models have impressive performance in many software engineering tasks. However, their effectiveness degrades when symbols are missing or not informative. The reason is that the model may not learn …
Language ModelingLanguage ModellingMasked Language ModelingPunctuation Restoration Improves Structure Understanding Without Supervision
Unsupervised learning objectives like autoregressive and masked language modeling constitute a significant part in producing pre-trained representations that perform various downstream applications from natural language …
ChunkingLanguage ModelingLanguage ModellingLarge Language Model+7VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation
Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate through realistic 3D outdoor environments based on natural language instructions. The performance of existing VLN methods is limited by insuffici…
Language ModelingLanguage ModellingMasked Language ModelingNavigate+1Arabic Synonym BERT-based Adversarial Examples for Text Classification
Text classification systems have been proven vulnerable to adversarial text examples, modified versions of the original text examples that are often unnoticed by human eyes, yet can force text classification models to al…
Adversarial TextLanguage ModelingLanguage ModellingMasked Language Modeling+2Interpretation of Intracardiac Electrograms Through Textual Representations
Understanding the irregular electrical activity of atrial fibrillation (AFib) has been a key challenge in electrocardiography. For serious cases of AFib, catheter ablations are performed to collect intracardiac electrogr…
Language ModelingLanguage ModellingMasked Language ModelingHow Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?
Recent breakthroughs in scale have enabled the emergence of powerful generative language models, and the ability to fine-tune these models on various tasks by casting them into prompts or instructions. In this landscape,…
ClassificationDomain Adaptationdomain classificationLanguage Modelling+3BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining
BERT (Bidirectional Encoder Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have main…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+1Endowing Protein Language Models with Structural Knowledge
Understanding the relationships between protein sequence, structure and function is a long-standing biological challenge with manifold implications from drug design to our understanding of evolution. Recently, protein la…
Drug DesignLanguage ModelingLanguage ModellingMasked Language Modeling+2Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling
Pre-trained language models based on masked language modeling (MLM) excel in natural language understanding (NLU) tasks. While fine-tuned MLM-based encoders consistently outperform causal language modeling decoders of co…
Causal Language ModelingDecoderLanguage ModelingLanguage Modelling+2Automated Scoring of Clinical Patient Notes using Advanced NLP and Pseudo Labeling
Clinical patient notes are critical for documenting patient interactions, diagnoses, and treatment plans in medical practice. Ensuring accurate evaluation of these notes is essential for medical education and certificati…
Language ModelingLanguage ModellingMasked Language ModelingA character-based steganography using masked language modeling
In this study, a steganography method based on BERT transformer model is proposed for hiding text data in cover text. The aim is to hide information by replacing specific words within the text using BERT’s masked languag…
Language ModelingLanguage ModellingMasked Language ModelingSemantic Similarity+1Iterative Mask Filling: An Effective Text Augmentation Method Using Masked Language Modeling
Data augmentation is an effective technique for improving the performance of machine learning models. However, it has not been explored as extensively in natural language processing (NLP) as it has in computer vision. In…
Data Augmentationfill-maskFill MaskLanguage Modeling+5Efficient Parallel Audio Generation using Group Masked Language Modeling
We present a fast and high-quality codec language model for parallel audio generation. While SoundStorm, a state-of-the-art parallel audio generation model, accelerates inference speed compared to autoregressive models, …
Audio GenerationComputational EfficiencyLanguage ModelingLanguage Modelling+1MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past half-decade since BERT first rose to promin…
GPULanguage ModelingLanguage ModellingMasked Language ModelingHCDIR: End-to-end Hate Context Detection, and Intensity Reduction model for online comments
Warning: This paper contains examples of the language that some people may find offensive. Detecting and reducing hateful, abusive, offensive comments is a critical and challenging task on social media. Moreover, few stu…
Hate Speech DetectionLanguage ModelingLanguage ModellingMasked Language ModelingECAMP: Entity-centered Context-aware Medical Vision Language Pre-training
Despite significant advancements in medical vision-language pre-training, existing methods have largely overlooked the inherent entity-specific context within radiology reports and the complex cross-modality contextual r…
Language ModelingLanguage ModellingLarge Language ModelMasked Language Modeling+1Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data
In this work we introduce Labrador, a pre-trained Transformer model for laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million lab test results from electronic health records (EHRs) and evaluated …
Language ModelingLanguage ModellingMasked Language ModelingTransfer Learning