Evaluating Multilingual BERT for Estonian

2020-10-01 · Claudia Kittask, Kirill Milintsevich, Kairit Sirts

Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet available. However, there exist several multilingual BERT models that can handle multiple languages simultaneously and that have been trained also on Estonian data. In this paper, we evaluate four multilingual models -- multilingual BERT, multilingual distilled BERT, XLM and XLM-RoBERTa -- on several NLP tasks including POS and morphological tagging, NER and text classification. Our aim is to establish a comparison between these multilingual BERT models and the existing baseline neural models for these tasks. Our results show that multilingual BERT models can generalise well on different Estonian NLP tasks outperforming all baselines models for POS and morphological tagging and text classification, and reaching the comparable level with the best baseline for NER, with XLM-RoBERTa achieving the highest results compared with other multilingual models.

📄 PDF Abstract BibTeX arXiv:2010.00454

Code (0)

등록된 구현이 없습니다.

Tasks

Morphological TaggingNERPOStext-classification Text Classification

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

XLM XLM is a Transformer based architecture that is pre-trained using one of three language modelling objectives: 1. Causal…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

EstBERT: A Pretrained Language-Specific BERT for Estonian

2020-11-09 · NoDaLiDa 2021 5 · Hasan Tanvir, Claudia Kittask, Sandra Eiche, Kairit Sirts

This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines…

Morphological Tagging named-entity-recognition Named Entity Recognition Named Entity Recognition (NER)+3

Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages

2021-12-20 · Matej Ulčar, Marko Robnik-Šikonja

Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. While studies have shown that monolingual models produce better results than multilingual models, the training dataset…

Dependency Parsing named-entity-recognition Named Entity Recognition Named Entity Recognition (NER)+1

Evaluating Transferability of BERT Models on Uralic Languages

2021-09-13 · ACL (IWCLUL) 2021 9 · Judit Ács, Dániel Lévai, András Kornai

Transformer-based language models such as BERT have outperformed previous models on a large number of English benchmarks, but their evaluation is often limited to English or a small number of well-resourced languages. In…

Hyperparameter Optimization NER POS

FinEst BERT and CroSloEngual BERT: less is more in multilingual models

2020-06-14 · Matej Ulčar, Marko Robnik-Šikonja

Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. The research has been mostly focused on English language, though. While massively multilingual models exist, studies h…

Dependency Parsing NER POS POS Tagging+1

Introducing OmniGEC: A Silver Multilingual Dataset for Grammatical Error Correction

2025-09-18 · Roman Kovalchuk, Mariana Romanyshyn, Petro Ivaniuk arxiv

In this paper, we introduce OmniGEC, a collection of multilingual silver-standard datasets for the task of Grammatical Error Correction (GEC), covering eleven languages: Czech, English, Estonian, German, Greek, Icelandic…

Grammatical Error Correction