Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution
In this paper, we develop bilingual transfer learning approaches to improve Arabic coreference resolution by leveraging additional English annotation via bilingual or multilingual pre-trained transformers. We show that bilingual transfer learning improves the strong transformer-based neural coreference models by 2-4 F1. We also systemically investigate the effectiveness of several pre-trained transformer models that differ in training corpora, languages covered, and model capacity. Our best model achieves a new state-of-the-art performance of 64.55 F1 on the Arabic OntoNotes dataset. Our code is publicly available at https://github.com/bnmin/arabic_coref.
Code (1)
Tasks
coreference-resolutionCoreference ResolutionTransfer LearningSimilar Papers 제목 키워드 기반
An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on …
Cross-Lingual TransferLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+8From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models
Arabic Language Models (LMs) are pretrained predominately on Modern Standard Arabic (MSA) and are expected to transfer to its dialects. While MSA as the standard written variety is commonly used in formal settings, peopl…
Cross-Lingual TransferExploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
Without any explicit cross-lingual training data, multilingual language models can achieve cross-lingual transfer. One common way to improve this transfer is to perform realignment steps before fine-tuning, i.e., to trai…
Cross-Lingual TransferPOSPOS TaggingXLM-RBiMediX: Bilingual Medical Mixture of Experts LLM
In this paper, we introduce BiMediX, the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic. Our model facilitates a wide range of medical interactions in English …
Mixture-of-ExpertsMultiple-choiceOpen-Ended Question AnsweringQuestion AnsweringBilingual Adaptation of Monolingual Foundation Models
We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama …
Continual PretrainingCross-Lingual TransferLanguage ModelingLanguage Modelling+1