When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer
While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks, there is a lack of consensus in the community as to what shared properties between languages enable such transfer. Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects. In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax. Among other things, our experiments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., R=0.94 on the task of NLI). Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.
Code (2)
Tasks
Cross-Lingual TransferSimilar Papers 제목 키워드 기반
When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer
While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks, there is a lack of consensus in the community as to what shared properties betwe…
Cross-Lingual TransferHerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
BERT-based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state-of-the-art results. Therefore, the NLP community conducts extensive research on understandi…
Language ModelingLanguage ModellingMixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi
One of the most popular downstream tasks in the field of Natural Language Processing is text classification. Text classification tasks have become more daunting when the texts are code-mixed. Though they are not exposed …
Language ModelingLanguage Modellingtext-classificationText Classification+1Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval
Existing approaches for image-to-recipe retrieval have the implicit assumption that a food image can fully capture the details textually documented in its recipe. However, a food image only reflects the visual outcome of…
Representation LearningA Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing…
Cross-Lingual TransferTranslationWord Embeddings