Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks
Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this paper, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with meta-learning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferMeta-LearningSimilar Papers 제목 키워드 기반
Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgish
Cross-lingual transfer has become a central paradigm for extending natural language processing (NLP) technologies to low-resource languages. By leveraging supervision from high-resource languages, multilingual language m…
Cross-Lingual TransferCross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources
Training a POS tagging model with crosslingual transfer learning usually requires linguistic knowledge and resources about the relation between the source language and the target language. In this paper, we introduce a c…
Cross-Lingual TransferLanguage ModelingLanguage ModellingNamed Entity Recognition (NER)+6On the Cross-lingual Transferability of Contextualized Sense Embeddings
In this paper we analyze the extent to which contextualized sense embeddings, i.e., sense embeddings that are computed based on contextualized word embeddings, are transferable across languages.To this end, we compiled a…
Word EmbeddingsWord Sense DisambiguationMT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction
Cross-lingual open information extraction aims to extract structured information from raw text across multiple languages. Previous work uses a shared cross-lingual pre-trained model to handle the different languages but …
Cross-Lingual TransferLanguage ModellingLarge Language ModelOpen Information ExtractionOrthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-s…
Cross-Lingual TransferNERPOSPOS Tagging+1