Spanish Pre-trained BERT Model and Evaluation Data
The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.
Code (2)
Tasks
Language ModelingLanguage ModellingmodelSimilar Papers 제목 키워드 기반
RoBERTuito: a pre-trained language model for social media text in Spanish
Since BERT appeared, Transformer language models and transfer learning have become state-of-the-art for Natural Language Understanding tasks. Recently, some works geared towards pre-training specially-crafted models for …
Language ModelingLanguage ModellingNatural Language UnderstandingTransfer LearningMarIA: Spanish Language Models
This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large…
Extractive Question-AnsweringQuestion AnsweringSequence-to-Sequence Spanish Pre-trained Language Models
In recent years, significant advancements in pre-trained language models have driven the creation of numerous non-English language variants, with a particular emphasis on encoder-only and decoder-only architectures. Whil…
DecoderGenerative Question AnsweringNatural Language UnderstandingQuestion Answering+1MEL: Legal Spanish Language Model
Legal texts, characterized by complex and specialized terminology, present a significant challenge for Language Models. Adding an underrepresented language, such as Spanish, to the mix makes it even more challenging. Whi…
Language ModelingLanguage ModellingmodelEvaluation Benchmarks for Spanish Sentence Representations
Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, t…
Language ModelingLanguage ModellingSentence