Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew
Pre-trained language models (PLMs) have shown remarkable successes in acquiring a wide range of linguistic knowledge, relying solely on self-supervised training on text streams. Nevertheless, the effectiveness of this language-agnostic approach has been frequently questioned for its sub-optimal performance when applied to morphologically-rich languages (MRLs). We investigate the hypothesis that incorporating explicit morphological knowledge in the pre-training phase can improve the performance of PLMs for MRLs. We propose various morphologically driven tokenization methods enabling the model to leverage morphological cues beyond raw text. We pre-train multiple language models utilizing the different methods and evaluate them on Hebrew, a language with complex and highly ambiguous morphology. Our experiments show that morphologically driven tokenization demonstrates improved results compared to a standard language-agnostic tokenization, on a benchmark of both semantic and morphologic tasks. These findings suggest that incorporating morphological knowledge holds the potential for further improving PLMs for morphologically rich languages.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Explicitly modeling case improves neural dependency parsing
Neural dependency parsing models that compose word representations from characters can presumably exploit morphosyntax when making attachment decisions. How much do they know about morphology? We investigate how well the…
Dependency ParsingMulti-Task LearningWhat do character-level models learn about morphology? The case of dependency parsing
When parsing morphologically-rich languages with neural models, it is beneficial to model input at the character level, and it has been claimed that this is because character-level models learn morphology. We test these …
Dependency ParsingMorphological AnalysisBetter Character Language Modeling Through Morphology
We incorporate morphological supervision into character language models (CLMs) via multitasking and show that this addition improves bits-per-character (BPC) performance across 24 languages, even when the morphology data…
Language ModelingLanguage ModellingUnderstanding and Improving Morphological Learning in the Neural Machine Translation Decoder
End-to-end training makes the neural machine translation (NMT) architecture simpler, yet elegant compared to traditional statistical machine translation (SMT). However, little is known about linguistic patterns of morpho…
DecoderMachine TranslationMulti-Task LearningNMT+1Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modeling
Morphological tagging is challenging for morphologically rich languages due to the large target space and the need for more training data to minimize model sparsity. Dialectal variants of morphologically rich languages s…
Morphological TaggingTransfer Learning