Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All
We present a new pre-trained language model (PLM) for modern Hebrew, termed AlephBERTGimmel, which employs a much larger vocabulary (128K items) than standard Hebrew PLMs before. We perform a contrastive analysis of this model against all previous Hebrew PLMs (mBERT, heBERT, AlephBERT) and assess the effects of larger vocabularies on task performance. Our experiments show that larger vocabularies lead to fewer splits, and that reducing splits is better for model performance, across different tasks. All in all this new model achieves new SOTA on all available Hebrew benchmarks, including Morphological Segmentation, POS Tagging, Full Morphological Analysis, NER, and Sentiment Analysis. Subsequently we advocate for PLMs that are larger not only in terms of number of layers or training data, but also in terms of their vocabulary. We release the new model publicly for unrestricted use.
Code (0)
등록된 구현이 없습니다.
Tasks
AllLanguage ModelingLanguage ModellingMorphological AnalysisNamed Entity Recognition (NER)NERPOSPOS TaggingSentiment AnalysisSimilar Papers 제목 키워드 기반
Simple, Fast Noise-Contrastive Estimation for Large RNN Vocabularies
Strategies for Training Large Vocabulary Neural Language Models
Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney. At the same time, neural language models are gaining popularity for …
Machine Translationspeech-recognitionSpeech RecognitionTranslationCMV-BERT: Contrastive multi-vocab pretraining of BERT
In this work, we represent CMV-BERT, which improves the pretraining of a language model via two ingredients: (a) contrastive learning, which is well studied in the area of computer vision; (b) multiple vocabularies, one …
Contrastive LearningLanguage ModelingLanguage ModellingSentenceFeature-based Decipherment for Large Vocabulary Machine Translation
Orthographic similarities across languages provide a strong signal for probabilistic decipherment, especially for closely related language pairs. The existing decipherment models, however, are not well-suited for exploit…
DeciphermentMachine TranslationTranslationBeing Generous with Sub-Words towards Small NMT Children
In the context of under-resourced neural machine translation (NMT), transfer learning from an NMT model trained on a high resource language pair, or from a multilingual NMT (M-NMT) model, has been shown to boost performa…
Machine TranslationNMTTransfer LearningTranslation