Papers Causal Language Modeling
“Causal Language Modeling” 태그가 달린 논문 52편 · 필터 해제
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
Transformer models face scalability challenges in causal language modeling (CLM) due to inefficient memory allocation for growing key-value (KV) caches, which strains compute and storage resources. Existing methods like …
Causal Language ModelingInstruction FollowingLanguage ModelingLanguage ModellingGRITHopper: Decomposition-Free Multi-Hop Dense Retrieval
Decomposition-based multi-hop retrieval methods rely on many autoregressive steps to break down complex queries, which breaks end-to-end differentiability and is computationally expensive. Decomposition-free methods tack…
Causal Language ModelingLanguage ModelingLanguage ModellingRetrievalTrojan Detection Through Pattern Recognition for Large Language Models
Trojan backdoors can be injected into large language models at various stages, including pretraining, fine-tuning, and in-context learning, posing a significant threat to the model's alignment. Due to the nature of causa…
Causal Language ModelingIn-Context LearningLanguage ModelingLanguage ModellingTowards the Anonymization of the Language Modeling
Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when pre-trained models fine-tuned and spe…
Causal Language ModelingLanguage ModelingLanguage ModellingPrivacy PreservingPreference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models
Alignment, endowing a pre-trained Large language model (LLM) with the ability to follow instructions, is crucial for its real-world applications. Conventional supervised fine-tuning (SFT) methods formalize it as causal l…
Causal Language ModelingLanguage ModelingLanguage ModellingLarge Language ModelAntLM: Bridging Causal and Masked Language Models
Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each p…
Causal Language ModelingDecoderLanguage ModelingLanguage Modelling+1Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning
Large language models (LLMs) have revolutionized the field of natural language processing with their impressive reasoning and question-answering capabilities. However, these models are sometimes prone to generating credi…
Causal Language ModelingLanguage ModelingLanguage ModellingQuestion Answering+2ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation
We introduce ElastiFormer, a post-training technique that adapts pretrained Transformer models into an elastic counterpart with variable inference time compute. ElastiFormer introduces small routing modules (as low as .0…
Causal Language ModelingLanguage ModelingLanguage ModellingGPT or BERT: why not both?
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transforme…
Causal Language ModelingLanguage ModelingLanguage ModellingMasked Language ModelingInterpretable Language Modeling via Induction-head Ngram Models
Recent large language models (LLMs) have excelled across a wide range of tasks, but their use in high-stakes and compute-limited settings has intensified the demand for interpretability and efficiency. We address this ne…
Causal Language ModelingHuman fMRI response predictionLanguage ModelingLanguage ModellingA Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers
Many real-world applications of tabular data involve using historic events to predict properties of new ones, for example whether a credit card transaction is fraudulent or what rating a customer will assign a product on…
Causal Language ModelingLanguage ModelingLanguage ModellingMissing ValuesQuAILoRA: Quantization-Aware Initialization for LoRA
QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively impact model performance after fine-tun…
Causal Language ModelingGPULanguage ModelingLanguage Modelling+2Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles
Causal language modeling using the Transformer architecture has yielded remarkable capabilities in Large Language Models (LLMs) over the last few years. However, the extent to which fundamental search and reasoning capab…
Causal Language ModelingLanguage ModelingLanguage ModellingLogical SequenceGenerating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling
The vast amount of available medical records has the potential to improve healthcare and biomedical research. However, privacy restrictions make these data accessible for internal use only. Recent works have addressed th…
Causal Language ModelingDe-identificationDiversityLanguage Modeling+4N-gram Prediction and Word Difference Representations for Language Modeling
Causal language modeling (CLM) serves as the foundational framework underpinning remarkable successes of recent large language models (LLMs). Despite its success, the training approach for next word prediction poses a po…
Causal Language ModelingLanguage ModelingLanguage ModellingMachine Translation+4Masked Mixers for Language Generation and Retrieval
Attention mechanisms that confer selective focus on a strict subset of input elements are nearly ubiquitous in language models today. We posit there to be downside to the use of attention: most information present in the…
Causal Language ModelingRetrievalText GenerationNovel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning
Teaching new information to pre-trained large language models (PLM) is a crucial but challenging task. Model adaptation techniques, such as fine-tuning and parameter-efficient training have been shown to store new facts …
Causal Language ModelingContinual LearningLanguage ModelingLanguage Modelling+2Predictability and Causality in Spanish and English Natural Language Generation
In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparitie…
Causal Language ModelingLanguage ModelingLanguage ModellingText GenerationConditional Language Learning with Context
Language models can learn sophisticated language understanding skills from fitting raw text. They also unselectively learn useless corpus statistics and biases, especially during finetuning on domain-specific corpora. In…
Causal Language ModelingLanguage ModelingLanguage ModellingLifelong learningUnderstanding Token Probability Encoding in Output Embeddings
In this paper, we investigate the output token probability information in the output embedding of language models. We find an approximate common log-linear encoding of output token probabilities within the output embeddi…
Causal Language ModelingLanguage ModelingLanguage Modelling