paper-with-me

홈 › Papers

mT5: A massively multilingual pre-trained text-to-text transformer

2020-10-22 · NAACL 2021 4 · Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

📄 PDF Abstract BibTeX arXiv:2010.11934

Code (8)

google-research/multilingual-t5 공식 구현 tf
KoshiroSato/Flask_NLP_App
KoshiroSato/Simple_Transformers_mT5_finetuning
MorenoLaQuatra/bart-it pytorch
google-research/byt5 tf
huggingface/transformers pytorch
manshri/tesum pytorch
pwc-1/Paper-5/tree/main/mt5 mindspore

Tasks

Common Sense ReasoningNatural Language InferenceQuestion AnsweringReading ComprehensionTranslation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
mT5 설명 없음
Attention 설명 없음
Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Similar Papers 제목 키워드 기반

A General-Purpose Multilingual Document Encoder

2023-05-11 · Onur Galoğlu, Robert Litschko, Goran Glavaš

Massively multilingual pretrained transformers (MMTs) have tremendously pushed the state of the art on multilingual NLP and cross-lingual transfer of NLP models in particular. While a large body of work leveraged MMTs to…

Cross-Lingual TransferDocument ClassificationLong-range modelingMultilingual NLP+2

Massively Multilingual Lexical Specialization of Multilingual Transformers

2022-08-01 · Tommaso Green, Simone Paolo Ponzetto, Goran Glavaš

While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-…

Bilingual Lexicon InductionRetrievalSentenceSentence Retrieval+3

Sliced at SemEval-2022 Task 11: Bigger, Better? Massively Multilingual LMs for Multilingual Complex NER on an Academic GPU Budget

2022-07-01 · SemEval (NAACL) 2022 7 · Barbara Plank

Massively multilingual language models (MMLMs) have become a widely-used representation method, and multiple large MMLMs were proposed in recent years. A trend is to train MMLMs on larger text corpora or with more layers…

GPUNERXLM-R

Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages

2025-08-12 · Imalsha Puranegedara, Themira Chathumina, Nisal Ranathunga, Nisansa de Silva 외 arxiv

Large Language Models (LLMs) excel in English, but their performance degrades significantly on low-resource languages (LRLs) due to English-centric training. While methods like LangBridge align LLMs with multilingual enc…

News Classification

Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

2019-06-04 · ACL 2019 7 · Benjamin Heinzerling, Michael Strube

Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack …

Multilingual Named Entity RecognitionMultilingual NLPnamed-entity-recognitionNamed Entity Recognition+2