Papers Winogrande
“Winogrande” 태그가 달린 논문 26편 · 필터 해제
Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma2
This technical report describes an experiment on autoregressive pre-training of Gemma2 2 billion parameter large language model (LLM) with 10\% on the Lithuanian language component of CulturaX from the point of view of c…
ARCBelebeleContinual LearningGSM8K+7More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
Aligning large language models (LLMs) with human values is an increasingly critical step in post-training. Direct Preference Optimization (DPO) has emerged as a simple, yet effective alternative to reinforcement learning…
ARCHellaSwagMMLUSafety Alignment+2WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization
In this study, we take a closer look at how Winograd schema challenges can be used to evaluate common sense reasoning in LLMs. Specifically, we evaluate generative models of different sizes on the popular WinoGrande benc…
Common Sense ReasoningMemorizationWinograndeObliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Models
Recent copyright agreements between AI companies and content creators underscore the need for fine-grained control over language models' ability to reproduce copyrighted text. Existing defenses-ranging from aggressive un…
HellaSwagMemorizationMMLUTruthfulQA+1Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments
Large Language Models (LLMs) have shown remarkable performance across various tasks, yet significant disparities remain for non-English languages, and especially native African languages. This paper addresses these dispa…
Clinical KnowledgeCollege MedicineCross-Lingual TransferMMLU+2PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches
As large language models (LLMs) increasingly shape the AI landscape, fine-tuning pretrained models has become more popular than in the pre-LLM era for achieving optimal performance in domain-specific tasks. However, pret…
GPUGSM8Kparameter-efficient fine-tuningQuestion Answering+1Judgment of Thoughts: Courtroom of the Binary Logical Reasoning in Large Language Models
This paper proposes a novel prompt engineering technique called Judgment of Thought (JoT) that is specifically tailored for binary logical reasoning tasks. JoT employs three roles$\unicode{x2014}$lawyer, prosecutor, and …
Fake News DetectionLanguage ModelingLanguage ModellingLarge Language Model+4$\texttt{metabench}$ -- A Sparse Benchmark to Measure General Ability in Large Language Models
Large Language Models (LLMs) vary in their abilities on a range of tasks. Initiatives such as the $\texttt{Open LLM Leaderboard}$ aim to quantify these differences with several large benchmarks (sets of test items to whi…
ARCGSM8KHellaSwagMMLU+2Promises, Outlooks and Challenges of Diffusion Language Modeling
The modern autoregressive Large Language Models (LLMs) have achieved outstanding performance on NLP benchmarks, and they are deployed in the real world. However, they still suffer from limitations of the autoregressive t…
ARCHellaSwagLanguage ModelingLanguage Modelling+1LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for la…
GSM8KHellaSwagHumanEvalMath+6LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significant roadblock to large-scale training. Fo…
GPUGSM8KLanguage ModelingLanguage Modelling+4Who's Harry Potter? Approximate Unlearning in LLMs
Large language models (LLMs) are trained on massive internet corpora that often contain copyrighted content. This poses legal and ethical challenges for the developers and users of these models, as well as the original a…
ARCGPUHellaSwagLanguage Modelling+1Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
Recent work on explainable NLP has shown that few-shot prompting can enable large pretrained language models (LLMs) to generate grammatical and factual natural language explanations for data labels. In this work, we stud…
WinograndeOn Curriculum Learning for Commonsense Reasoning
Commonsense reasoning tasks follow a standard paradigm of finetuning pretrained language models on the target task data, where samples are introduced to the model in a random order during training. However, recent resear…
HellaSwagLearning-To-RankNatural Language UnderstandingQuestion Answering+1A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models
We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error…
Constituency ParsingGrammatical Error Detectionnamed-entity-recognitionNamed Entity Recognition+3ST-MoE: Designing Stable and Transferable Sparse Expert Models
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more ca…
ARCCommon Sense ReasoningCoreference ResolutionDecoder+8An Application of Pseudo-Log-Likelihoods to Natural Language Scoring
Language models built using semi-supervised machine learning on large corpora of natural language have very quickly enveloped the fields of natural language generation and understanding. In this paper we apply a zero-sho…
Common Sense ReasoningGPULanguage ModelingLanguage Modelling+3A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models
We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error…
Constituency ParsingGrammatical Error Detectionnamed-entity-recognitionNamed Entity Recognition+3Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup
Training a model to provide natural language explanations (NLEs) for its predictions usually requires the acquisition of task-specific NLEs, which is time- and resource-consuming. A potential solution is the few-shot out…
Natural Language InferenceTransfer LearningWinograndeScaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis o…
Abstract AlgebraAnachronismsAnalogical SimilarityAnalytic Entailment+143