Evaluation of Transfer Learning for Polish with a text-to-text model

2021-10-16 · ACL ARR October 2021 10 · Anonymous

We present polT - a general purpose text-to-text model for Polish that can be fine-tuned on a variety on Natural Language Processing (NLP) tasks with a single training objective. Unsupervised denoising pre-training is performed efficiently by initializing the model weights with multi-lingual T5 (mT5) counterpart. We evaluate performance of polT, mT5, Polish BART (plBART) and Polish GPT-2 (papuGaPT2) on diverse downstream tasks such as: text-to-text KLEJ benchmark, en-pl machine translation, question answering and summarization. The polT scores top on all of this tasks except summarization where plBART is best. In general (except summarization), the larger the model the better the results. The encoder-decoder architectures prove to be better than decoder-only equivalent. Additionally, since summarization and question answering lack benchmark datasets for Polish language we describe in detail their construction and will make them publicly available.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Denoising Machine Translation Question Answering Transfer Learning Translation

Methods 이 논문이 사용한 방법론

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Adafactor Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of…

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

SentencePiece 설명 없음

Evaluation of Transfer Learning for Polish with a Text-to-Text Model

2022-05-18 · LREC 2022 6 · Aleksandra Chrabrowa, Łukasz Dragan, Karol Grzegorczyk, Dariusz Kajtoch 외

We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, a…

Decoder Denoising Question Answering Transfer Learning

Two Approaches to Diachronic Normalization of Polish Texts

2024-02-02 · Kacper Dudzic, Filip Graliński, Krzysztof Jassem, Marek Kubis 외

This paper discusses two approaches to the diachronic normalization of Polish texts: a rule-based solution that relies on a set of handcrafted patterns, and a neural normalization model based on the text-to-text transfer…

Keyword Extraction from Short Texts with a Text-To-Text Transfer Transformer

2022-09-28 · Piotr Pęzik, Agnieszka Mikołajczyk-Bareła, Adam Wawrzyński, Bartłomiej Nitoń 외

The paper explores the relevance of the Text-To-Text Transfer Transformer language model (T5) for Polish (plT5) to the task of intrinsic and extrinsic keyword extraction from short text passages. The evaluation is carrie…

Keyword Extraction Language Modeling Language Modelling

The Polish Summaries Corpus

2014-05-01 · LREC 2014 5 · Maciej Ogrodniczuk, Mateusz Kope{\'c}

This article presents the Polish Summaries Corpus, a new resource created to support the development and evaluation of the tools for automated single-document summarization of Polish. The Corpus contains a large number o…

Articles Document Summarization

Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing

2025-02-21 · Shoumik Saha, Soheil Feizi

The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content u…

Text Generation