Bilingual Language Modeling, A transfer learning technique for Roman Urdu
Pretrained language models are now of widespread use in Natural Language Processing. Despite their success, applying them to Low Resource languages is still a huge challenge. Although Multilingual models hold great promise, applying them to specific low-resource languages e.g. Roman Urdu can be excessive. In this paper, we show how the code-switching property of languages may be used to perform cross-lingual transfer learning from a corresponding high resource language. We also show how this transfer learning technique termed Bilingual Language Modeling can be used to produce better performing models for Roman Urdu. To enable training and experimentation, we also present a collection of novel corpora for Roman Urdu extracted from various sources and social networking sites, e.g. Twitter. We train Monolingual, Multilingual, and Bilingual models of Roman Urdu - the proposed bilingual model achieves 23% accuracy compared to the 2% and 11% of the monolingual and multilingual models respectively in the Masked Language Modeling (MLM) task.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferLanguage ModelingLanguage ModellingMasked Language ModelingTransfer LearningSimilar Papers 제목 키워드 기반
RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning
In recent studies, it has been shown that Multilingual language models underperform their monolingual counterparts. It is also a well-known fact that training and maintaining monolingual models for each language is a cos…
Cross-Lingual TransferTransfer LearningBilingual Dictionary-based Language Model Pretraining for Neural Machine Translation
Recent studies have demonstrated a perceivable improvement on the performance of neural machine translation by applying cross-lingual language model pretraining (Lample and Conneau, 2019), especially the Translation Lang…
Language ModelingLanguage ModellingMachine TranslationTranslationRomanLens: Latent Romanization and its role in Multilinguality in LLMs
Large Language Models (LLMs) exhibit remarkable multilingual generalization despite being predominantly trained on English-centric corpora. A fundamental question arises: how do LLMs achieve such robust multilingual capa…
Language ModelingLanguage ModellingThe Trilingual ALLEGRA Corpus: Presentation and Possible Use for Lexicon Induction
In this paper, we present a trilingual parallel corpus for German, Italian and Romansh, a Swiss minority language spoken in the canton of Grisons. The corpus called ALLEGRA contains press releases automatically gathered …
SentenceA language-independent and fully unsupervised approach to lexicon induction and part-of-speech tagging for closely related languages
In this paper, we describe our generic approach for transferring part-of-speech annotations from a resourced language towards an etymologically closely related non-resourced language, without using any bilingual (i.e., p…
Part-Of-Speech TaggingPOSTranslation