Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
Code (3)
Tasks
Abstract AlgebraAnachronismsAnalogical SimilarityAnalytic EntailmentAnatomyAstronomyBIG-bench Machine LearningBusiness EthicsCausal JudgmentCheckmate In OneClinical KnowledgeCode Line DescriptionsCollege BiologyCollege ChemistryCollege Computer ScienceCollege MathematicsCollege MedicineCollege PhysicsCommon Sense ReasoningComputer SecurityConceptual PhysicsCrash BlossomCrass AIDark Humor DetectionDate UnderstandingDisambiguation QADiscourse Marker PredictionEconometricsElectrical EngineeringElementary MathematicsEmotional IntelligenceEmpirical JudgmentsEnglish ProverbsEntailed PolarityEpistemic ReasoningEthicsEvaluating Information EssentialityFact CheckingFantasy ReasoningFEVER (2-way)FEVER (3-way)Figure Of Speech DetectionFormal Fallacies Syllogisms NegationFormal LogicGeneral KnowledgeGlobal FactsGRE Reading ComprehensionHellaSwagHigh School BiologyHigh School ChemistryHigh School Computer ScienceHigh School European HistoryHigh School GeographyHigh School Government and PoliticsHigh School MacroeconomicsHigh School MathematicsHigh School MicroeconomicsHigh School PhysicsHigh School PsychologyHigh School StatisticsHigh School US HistoryHigh School World HistoryHindu KnowledgeHuman AgingHuman Organs Senses Multiple ChoiceHuman SexualityHyperbatonIdentify Odd MetaporImplicaturesImplicit RelationsIntelligent CommunicationIntent RecognitionInternational LawIrony IdentificationJurisprudenceKnown UnknownsLAMBADALanguage ModelingLanguage ModellingLogical ArgsLogical FallaciesLogical Fallacy DetectionLogical ReasoningLogical SequenceLogic Grid PuzzleManagementMarketingMathematical InductionMathematical ReasoningMedical GeneticsMemorizationMetaphor BooleanMiscellaneousMisconceptionsMoral DisputesMoral PermissibilityMoral ScenariosMovie Dialog Same Or DifferentMovie Genre Recommendation SystemMovie RecommendationMultiple Choice Question Answering (MCQA)Multi-task Language UnderstandingNatural QuestionsNavigateNonsense Words GrammarNovel ConceptsNutritionOdd One OutPenguins In A TablePhilosophyPhrase RelatednessPhysical IntuitionPhysics MCPrehistoryPresuppositions As NLIProfessional AccountingProfessional LawProfessional MedicineProfessional PsychologyPublic RelationsQuestion AnsweringQuestion SelectionRACE-hRACE-mReading ComprehensionReasoning About Colored ObjectsRiddle SenseRuin NamesSarcasm DetectionSecurity StudiesSentence AmbiguitySentence CompletionSimilarities AbstractionSNARKSSociologySports UnderstandingStrategyQATemporal SequencesTimedialTriviaQAUnderstanding FablesUS Foreign PolicyVirologyWinograndeWinowhyWord Sense DisambiguationWorld ReligionsSimilar Papers 제목 키워드 기반
Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and discrepancies of scaling laws b…
Mixture-of-ExpertsScale Efficiently: Insights from Pretraining and Finetuning Transformers
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost w…
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost w…
Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named $\textbf{model steerin…
Contrastive LearningGeneralization BoundsmodelPerformance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference
Aligning future system design with the ever-increasing compute needs of large language models (LLMs) is undoubtedly an important problem in today's world. Here, we propose a general performance modeling methodology and w…
GPULanguage ModelingLanguage ModellingLarge Language Model