AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.
Code (1)
Tasks
Causal Language ModelingCommon Sense ReasoningCoreference ResolutionDecoderDenoisingFew-Shot LearningLanguage ModelingLanguage ModellingMachine TranslationNatural Language InferenceQuestion AnsweringWord Sense DisambiguationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging
We present LINGUIST, a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST), via fine-tuning AlexaTM 5B, a 5-billion-parameter multilingual sequence-to-sequence (seq2seq) model, on a fl…
intent-classificationIntent ClassificationLanguage ModelingLanguage Modelling+2CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing
A bottleneck to developing Semantic Parsing (SP) models is the need for a large volume of human-labeled training data. Given the complexity and cost of human annotation for SP, labeled data is often scarce, particularly …
Data AugmentationSemantic ParsingGeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning
The emergence of Large Language Models (LLMs) with capabilities like In-Context Learning (ICL) has ushered in new possibilities for data generation across various domains while minimizing the need for extensive data coll…
Extractive Question-AnsweringFew-Shot LearningIn-Context LearningMachine Translation+2A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of fre…
Decoderparameter-efficient fine-tuningFew-Shot Multilingual Open-Domain QA from 5 Examples
Recent approaches to multilingual open-domain question answering (MLODQA) have achieved promising results given abundant language-specific training data. However, the considerable annotation cost limits the application o…
Few-Shot LearningOpen-Domain Question AnsweringQuestion Answering