PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
In natural language processing (NLP), there is a need for more resources in Portuguese, since much of the data used in the state-of-the-art research is in other languages. In this paper, we pretrain a T5 model on the BrWac corpus, an extensive collection of web pages in Portuguese, and evaluate its performance against other Portuguese pretrained models and multilingual models on three different tasks. We show that our Portuguese pretrained models have significantly better performance over the original T5 models. Moreover, we demonstrate the positive impact of using a Portuguese vocabulary. Our code and models are available at https://github.com/unicamp-dl/PTT5.
Code (3)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Acoustic models of Brazilian Portuguese Speech based on Neural Transformers
An acoustic model, trained on a significant amount of unlabeled data, consists of a self-supervised learned speech representation useful for solving downstream tasks, perhaps after a fine-tuning of the model in the respe…
PeLLE: Encoder-based language models for Brazilian Portuguese based on open data
In this paper we present PeLLE, a family of large language models based on the RoBERTa architecture, for Brazilian Portuguese, trained on curated, open data from the Carolina corpus. Aiming at reproducible results, we de…
Image captioning for Brazilian Portuguese using GRIT model
This work presents the early development of a model of image captioning for the Brazilian Portuguese language. We used the GRIT (Grid - and Region-based Image captioning Transformer) model to accomplish this work. GRIT i…
Image CaptioningmodelFrom Brazilian Portuguese to European Portuguese
Brazilian Portuguese and European Portuguese are two varieties of the same language and, despite their close similarities, they exhibit several differences. However, there is a significant disproportion in the availabili…
SentenceTranslationEvaluating GPT-3.5 and GPT-4 on Grammatical Error Correction for Brazilian Portuguese
We investigate the effectiveness of GPT-3.5 and GPT-4, two large language models, as Grammatical Error Correction (GEC) tools for Brazilian Portuguese and compare their performance against Microsoft Word and Google Docs.…
Grammatical Error Correction