Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
BMFM-RNA: An Open Framework for Building and Evaluating Transcriptomic Foundation Models
Transcriptomic foundation models (TFMs) have recently emerged as powerful tools for analyzing gene expression in cells and tissues, supporting key tasks such as cell-type annotation, batch correction, and perturbation pr…
BenchmarkingLanguage ModelingLanguage ModellingMasked Language ModelingGeoRecon: Graph-Level Representation Learning for 3D Molecules via Reconstruction-Based Pretraining
The pretraining-and-finetuning paradigm has driven significant advances across domains, such as natural language processing and computer vision, with representative pretraining paradigms such as masked language modeling …
DenoisingLanguage ModelingLanguage ModellingMasked Language Modeling+3Diffusion Sequence Models for Enhanced Protein Representation and Generation
Proteins are fundamental to biology, executing diverse functions through complex physicochemical interactions, and they hold transformative potential across medicine, materials science, and environmental applications. Pr…
Language ModelingLanguage ModellingMasked Language ModelingProtein Design+1Masked Language Models are Good Heterogeneous Graph Generalizers
Heterogeneous graph neural networks (HGNNs) excel at capturing structural and semantic information in heterogeneous graphs (HGs), while struggling to generalize across domains and tasks. Recently, some researchers have t…
Graph LearningLanguage ModelingLanguage ModellingMasked Language ModelingImproving Low-Resource Morphological Inflection via Self-Supervised Objectives
Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world's languages. Surprisingly, they have not been explored much for c…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+1GigaAM: Efficient Self-Supervised Learner for Speech Recognition
Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language…
Automatic Speech RecognitionLanguage ModelingLanguage ModellingMasked Language Modeling+3HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
Inspired by the great success of Masked Language Modeling (MLM) in the natural language domain, the paradigm of self-supervised pre-training and fine-tuning has also achieved remarkable progress in the field of DNA seque…
Language ModelingLanguage ModellingMasked Language ModelingAnkh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations
Protein language models (PLMs) have emerged as powerful tools to detect complex patterns of protein sequences. However, the capability of PLMs to fully capture information on protein sequences might be limited by focusin…
DenoisingLanguage ModelingLanguage ModellingMasked Language ModelingADALog: Adaptive Unsupervised Anomaly detection in Logs with Self-attention Masked Language Model
Modern software systems generate extensive heterogeneous log data with dynamic formats, fragmented event sequences, and varying temporal patterns, making anomaly detection both crucial and challenging. To address these c…
Anomaly DetectionLanguage ModelingLanguage ModellingLog Parsing+2CreoPep: A Universal Deep Learning Framework for Target-Specific Peptide Design and Optimization
Target-specific peptides, such as conotoxins, exhibit exceptional binding affinity and selectivity toward ion channels and receptors. However, their therapeutic potential remains underutilized due to the limited diversit…
DiversityLanguage ModelingLanguage ModellingMasked Language ModelingCodeSSM: Towards State Space Models for Code Understanding
Although transformers are widely used for various code-specific tasks, they have some significant limitations. In this paper, we investigate State Space Models (SSMs) as a potential alternative to transformers for code u…
Clone DetectionLanguage ModelingLanguage ModellingMasked Language Modeling+3In-Context Learning can distort the relationship between sequence likelihoods and biological fitness
Language models have emerged as powerful predictors of the viability of biological sequences. During training these models learn the rules of the grammar obeyed by sequences of amino acids or nucleotides. Once trained, t…
In-Context LearningLanguage ModelingLanguage ModellingMasked Language ModelingLow-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models
As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. Transliteration between Urdu and its Romanize…
Information RetrievalLanguage ModelingLanguage ModellingMasked Language Modeling+2Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them
We investigate the use of LLM-generated data for continual pretraining of encoder models in specialized domains with limited training data, using the scientific domain of invasion biology as a case study. To this end, we…
Continual PretrainingLanguage ModelingLanguage ModellingMasked Language ModelingLakotaBERT: A Transformer-based Model for Low Resource Lakota Language
Lakota, a critically endangered language of the Sioux people in North America, faces significant challenges due to declining fluency among younger generations. This paper introduces LakotaBERT, the first large language m…
Language ModelingLanguage ModellingLarge Language ModelMasked Language ModelingShushing! Let's Imagine an Authentic Speech from the Silent Video
Vision-guided speech generation aims to produce authentic speech from facial appearance or lip motions without relying on auditory signals, offering significant potential for applications such as dubbing in filmmaking an…
cross-modal alignmentLanguage ModelingLanguage ModellingMasked Language ModelingASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuning
Assembly code analysis and comprehension play critical roles in applications like reverse engineering, yet they face substantial challenges due to low information density and a lack of explicit syntactic structures. Whil…
Code GenerationDecoderInstruction FollowingLanguage Modeling+2Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
Masked language modeling has become a widely adopted unsupervised technique to pre-train language models. However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically f…
Authorship AttributionLanguage ModelingLanguage ModellingMasked Language Modeling+4Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
Large Language Models (LLMs) are discovered to suffer from accurately retrieving key information. To address this, we propose Mask-Enhanced Autoregressive Prediction (MEAP), a simple yet effective training paradigm that …
DecoderInformation RetrievalLanguage ModelingLanguage Modelling+2Enabling Autoregressive Models to Fill In Masked Tokens
Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR models are inherently incapable of masked …
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling