Papers LAMBADA
“LAMBADA” 태그가 달린 논문 30편 · 필터 해제
Matryoshka Model Learning for Improved Elastic Student Models
Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple…
LAMBADAMathmodelAdaGC: Improving Training Stability for Large Language Model Pretraining
Large Language Models (LLMs) face increasing loss spikes during scaling, undermining training stability and final performance. While gradient clipping mitigates this issue, traditional global approaches poorly handle par…
LAMBADALanguage ModelingLanguage ModellingLarge Language ModelExplaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM
Contrastive decoding (CD) (Li et al., 2023) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. Although CD is applied to various LMs and domains to enhance open-ended tex…
LAMBADAText GenerationBeyond Autoregression: Fast LLMs via Self-Distillation Through Time
Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs…
Automated Theorem ProvingCode GenerationLAMBADANatural Language UnderstandingSpectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
Rapid advancements in GPU computational power has outpaced memory capacity and bandwidth growth, creating bottlenecks in Large Language Model (LLM) inference. Post-training quantization is the leading method for addressi…
GPULAMBADALanguage ModellingLarge Language Model+1SymBa: Symbolic Backward Chaining for Structured Natural Language Reasoning
To improve the performance and explainability of LLM-based natural language reasoning, structured reasoning can be applied to generate explicitly structured proofs. Among different methods for structured reasoning, we sp…
Arithmetic ReasoningGSM8KLAMBADAPIXAR: Auto-Regressive Language Modeling in Pixel Space
Recent work showed the possibility of building open-vocabulary large language models (LLMs) that directly operate on pixel representations. These models are implemented as autoencoders that reconstruct masked patches of …
DecoderLAMBADALanguage ModelingLanguage Modelling+3Concise and Organized Perception Facilitates Reasoning in Large Language Models
Exploiting large language models (LLMs) to tackle reasoning has garnered growing attention. It still remains highly challenging to achieve satisfactory results in complex logical problems, characterized by plenty of prem…
LAMBADAMathHeadless Language Models: Learning without Predicting with Contrastive Weight Tying
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability…
LAMBADAStay on topic with Classifier-Free Guidance
Classifier-Free Guidance (CFG) has recently emerged in text-to-image generation as a lightweight technique to encourage prompt-adherence in generations. In this work, we demonstrate that CFG can be used broadly as an inf…
Code GenerationCommon Sense ReasoningImage GenerationLAMBADA+8Inconsistencies in Masked Language Models
Learning to predict masked tokens in a sequence has been shown to be a helpful pretraining objective for powerful language models such as PaLM2. After training, such masked language models (MLMs) can provide distribution…
LAMBADAMMLULAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Remarkable progress has been made on automated reasoning with natural text, by using Language Models (LMs) and methods such as Chain-of-Thought and Selection-Inference. These techniques search for proofs in the forward d…
LAMBADALogical ReasoningLeveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling
In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Tran…
LAMBADALearning to ExecuteMachine TranslationTranslationTraining Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence …
AnachronismsAnalogical SimilarityAnalytic EntailmentCausal Judgment+69Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis o…
Abstract AlgebraAnachronismsAnalogical SimilarityAnalytic Entailment+143CoreLM: Coreference-aware Language Model Fine-Tuning
Language Models are the underpin of all modern Natural Language Processing (NLP) tasks. The introduction of the Transformers architecture has contributed significantly into making Language Modeling very effective across …
LAMBADALanguage ModelingLanguage ModellingmodelThe Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
Recent works have demonstrated great success in pre-training large-scale autoregressive language models on massive GPUs. To reduce the wall-clock training time, a common practice is to increase the batch size and learnin…
LAMBADAText GenerationE.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks
In the last decade, the field of Neural Language Modelling has witnessed enormous changes, with the development of novel models through the use of Transformer architectures. However, even these models struggle to model l…
LAMBADALanguage ModelingLanguage Modellingnamed-entity-recognition+2Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences
Attention is a commonly used mechanism in sequence processing, but it is of O(n^2) complexity which prevents its application to long sequences. The recently introduced neural Shuffle-Exchange network offers a computation…
LAMBADALanguage ModellingMusic TranscriptionNeural Shuffle-Exchange Networks - Sequence Processing in O(n log n) Time
A key requirement in sequence to sequence processing is the modeling of long range dependencies. To this end, a vast majority of the state-of-the-art models use attention mechanism which is of O(n^2) complexity that lead…
LAMBADALanguage ModelingLanguage ModellingMusic Generation+1