Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
It is now established that modern neural language models can be successfully trained on multiple languages simultaneously without changes to the underlying architecture. But what kind of knowledge is really shared among languages within these models? Does multilingual training mostly lead to an alignment of the lexical representation spaces or does it also enable the sharing of purely grammatical knowledge? In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors, using a variety of models and probing tasks. We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferSimilar Papers 제목 키워드 기반
Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?
Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, whereby it enables effective zero-shot cross-lingual transfer of syntactic knowledge. The transfer is more successful between some …
Cross-Lingual TransferDiversityZero-Shot Cross-Lingual TransferAssessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of cross-lingual transfer tasks. However, it remains unknown w…
Cross-Lingual TransferIncorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer
Unsupervised cross-lingual transfer involves transferring knowledge between languages without explicit supervision. Although numerous studies have been conducted to improve performance in such tasks by focusing on cross-…
Cross-Lingual TransferGraph Attentionnamed-entity-recognitionNamed Entity Recognition+6Finding Universal Grammatical Relations in Multilingual BERT
Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations a…
Cross-Lingual TransferLanguage ModelingLanguage ModellingZero-Shot Cross-Lingual TransferLearning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension
Multilingual pre-trained models are able to zero-shot transfer knowledge from rich-resource to low-resource languages in machine reading comprehension (MRC). However, inherent linguistic discrepancies in different langua…
Cross-Lingual TransferDisentanglementMachine Reading ComprehensionReading Comprehension+1