EstBERT: A Pretrained Language-Specific BERT for Estonian
This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines. Still, based on existing studies on other languages, a language-specific BERT model is expected to improve over the multilingual ones. We first describe the EstBERT pretraining process and then present the results of the models based on finetuned EstBERT for multiple NLP tasks, including POS and morphological tagging, named entity recognition and text classification. The evaluation results show that the models based on EstBERT outperform multilingual BERT models on five tasks out of six, providing further evidence towards a view that training language-specific BERT models are still useful, even when multilingual models are available.
Code (0)
등록된 구현이 없습니다.
Tasks
Morphological Taggingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)POStext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Comparison of Current Approaches to Lemmatization: A Case Study in Estonian
This study evaluates three different lemmatization approaches to Estonian -- Generative character-level models, Pattern-based word-level classification models, and rule-based morphological analysis. According to our expe…
ClassificationLemmatizationMorphological AnalysisKliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts
The increasing application of Natural Language Processing (NLP) in healthcare demands language models specifically attuned to the complexities of clinical language. This work introduces KliniskVestBERT, a suite of three …
Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages
Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. While studies have shown that monolingual models produce better results than multilingual models, the training dataset…
Dependency Parsingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1Evaluating Multilingual BERT for Estonian
Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet availabl…
Morphological TaggingNERPOStext-classification+1Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets
This study introduces an approach to Estonian text simplification using two model architectures: a neural machine translation model and a fine-tuned large language model (LLaMA). Given the limited resources for Estonian,…
Language ModelingLanguage ModellingLarge Language ModelMachine Translation+2