Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuray
Establishing the long-context capability of large vision-language models is crucial for video understanding, high-resolution image understanding, multi-modal agents and reasoning. We introduce Long-VITA, a simple yet eff…
4kGeneral KnowledgeGPULanguage Modeling+3SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling
In this paper, we propose "SoundSpring", a cutting-edge error-resilient audio transceiver that marries the robustness benefits of joint source-channel coding (JSCC) while also being compatible with current digital commun…
Audio CompressionLanguage ModelingLanguage ModellingMasked Language Modeling+1Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search
In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data. However, recent approaches encounter two principal chall…
Language ModelingLanguage ModellingMasked Language ModelingPerson Search+1Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach
The use of language models (LMs) has increased considerably in recent years, and the biases and stereotypes in training data that are reflected in the LM outputs are causing social problems. In this paper, inspired by th…
Language ModelingLanguage ModellingMasked Language ModelingTask ArithmeticA Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer
Language model approaches have recently been integrated into binary analysis tasks, such as function similarity detection and function signature recovery. These models typically employ a two-stage training process: pre-t…
Feature EngineeringLanguage ModelingLanguage ModellingMasked Language Modeling+1Small Languages, Big Models: A Study of Continual Training on Languages of Norway
Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern S\'ami. To address this iss…
Language ModelingLanguage ModellingMasked Language ModelingLeveraging Prompt Learning and Pause Encoding for Alzheimer's Disease Detection
Compared to other clinical screening techniques, speech-and-language-based automated Alzheimer's disease (AD) detection methods are characterized by their non-invasiveness, cost-effectiveness, and convenience. Previous s…
Alzheimer's Disease DetectionAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Language Modeling+5AntLM: Bridging Causal and Masked Language Models
Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each p…
Causal Language ModelingDecoderLanguage ModelingLanguage Modelling+1Mitigating Gender Bias in Contextual Word Embeddings
Word embeddings have been shown to produce remarkable results in tackling a vast majority of NLP related tasks. Unfortunately, word embeddings also capture the stereotypical biases that are prevalent in society, affectin…
Language ModelingLanguage ModellingMasked Language ModelingWord EmbeddingsCamemBERT 2.0: A Smarter French Language Model Aged to Perfection
French language models, such as CamemBERT, have been widely adopted across industries for natural language processing (NLP) tasks, with models like CamemBERT seeing over 4 million downloads per month. However, these mode…
Language ModelingLanguage ModellingMasked Language ModelingGPT or BERT: why not both?
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transforme…
Causal Language ModelingLanguage ModelingLanguage ModellingMasked Language ModelingLess is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies
Curriculum Learning has been a popular strategy to improve the cognitive plausibility of Small-Scale Language Models (SSLMs) in the BabyLM Challenge. However, it has not led to considerable improvements over non-curricul…
Language AcquisitionMasked Language ModelingAbrupt Learning in Transformers: A Case Study on Matrix Completion
Recent analysis on the training dynamics of Transformers has unveiled an interesting characteristic: the training loss plateaus for a significant number of training steps, and then suddenly (and sharply) drops to near--o…
Language ModelingLanguage ModellingLow-Rank Matrix CompletionMasked Language Modeling+1Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers
Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architec…
Drug DesignLanguage ModelingLanguage ModellingMamba+4Distributionally robust self-supervised learning for tabular data
Machine learning (ML) models trained using Empirical Risk Minimization (ERM) often exhibit systematic errors on specific subpopulations of tabular data, known as error slices. Learning robust representation in presence o…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+2DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models
Discrete diffusion models have achieved success in tasks like image generation and masked language modeling but face limitations in controlled content editing. We introduce DICE (Discrete Inversion for Controllable Editi…
Image GenerationLanguage ModelingLanguage ModellingMasked Language ModelingLecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERT
Logical errors in programming don't raise compiler alerts, making them hard to detect. These silent errors can disrupt a program's function or cause run-time issues. Their correction requires deep insight into the progra…
Language ModelingLanguage ModellingLarge Language ModelMasked Language ModelingEnhancing SPARQL Generation by Triplet-order-sensitive Pre-training
Semantic parsing that translates natural language queries to SPARQL is of great importance for Knowledge Graph Question Answering (KGQA) systems. Although pre-trained language models like T5 have achieved significant suc…
Graph Question AnsweringLanguage ModelingLanguage ModellingMasked Language Modeling+5FARM: Functional Group-Aware Representations for Small Molecules
We introduce Functional Group-Aware Representations for Small Molecules (FARM), a novel foundation model designed to bridge the gap between SMILES, natural language, and molecular graphs. The key innovation of FARM lies …
Contrastive LearningDrug DiscoveryLanguage ModelingLanguage Modelling+3SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics
Prompt-based fine-tuning has become an essential method for eliciting information encoded in pre-trained language models for a variety of tasks, including text classification. For multi-class classification tasks, prompt…
ClassificationLanguage ModelingLanguage ModellingMasked Language Modeling+3