Papers Multi-task Language Understanding
“Multi-task Language Understanding” 태그가 달린 논문 57편 · 필터 해제
Scaling Instruction-Finetuned Language Models
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a partic…
Coreference ResolutionCross-Lingual Question AnsweringMMLUMulti-task Language Understanding+2GLM-130B: An Open Bilingual Pre-trained Model
We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 (davinci) and unveil how model…
Language ModelingLanguage ModellingLong-Context Understandingmodel+2Atlas: Few-shot Learning with Retrieval Augmented Language Models
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive param…
Fact CheckingFew-Shot LearningInformation RetrievalLanguage Modeling+7Solving Quantitative Reasoning Problems with Language Models
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitati…
Arithmetic ReasoningLanguage ModelingLanguage ModellingLarge Language Model+3UL2: Unifying Language Learning Paradigms
Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents …
Arithmetic ReasoningCommon Sense ReasoningCoreference ResolutionIn-Context Learning+10GPT-NeoX-20B: An Open-Source Autoregressive Language Model
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best o…
Language ModelingLanguage ModellingMulti-task Language UnderstandingPaLM: Scaling Language Modeling with Pathways
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed t…
Auto DebuggingCode GenerationCommon Sense ReasoningCoreference Resolution+19Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence …
AnachronismsAnalogical SimilarityAnalytic EntailmentCausal Judgment+69Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis o…
Abstract AlgebraAnachronismsAnalogical SimilarityAnalytic Entailment+143Merging Models with Fisher-Weighted Averaging
Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation c…
Domain AdaptationMulti-task Language UnderstandingTransfer LearningEvaluating Large Language Models Trained on Code
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a ne…
Code GenerationHumanEvalLanguage ModelingLanguage Modelling+1Measuring Massive Multitask Language Understanding
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models mus…
Elementary MathematicsMulti-task Language UnderstandingMulti-Task LearningWorld KnowledgeLanguage Models are Few-Shot Learners
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this me…
answerability predictionArticlesCommon Sense ReasoningCoreference Resolution+14UnifiedQA: Crossing Format Boundaries With a Single QA System
Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc. This has led to format-specialized models, and even to an implicit division in the QA com…
Common Sense ReasoningLanguage ModelingLanguage ModellingMultiple-choice+3ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limita…
Common Sense ReasoningGPULinguistic AcceptabilityMultimodal Intent Recognition+6RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different s…
Common Sense ReasoningDocument Image ClassificationLanguage ModelingLanguage Modelling+14Language Models are Unsupervised Multitask Learners
Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate th…
Common Sense ReasoningCoreference ResolutionData-to-Text GenerationDialogue State Tracking+12