MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.
Code (2)
Tasks
Cross-Lingual TransferMeta-Learningnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Sentiment AnalysisTransfer LearningSimilar Papers 제목 키워드 기반
MetaXLR -- Mixed Language Meta Representation Transformation for Low-resource Cross-lingual Learning based on Multi-Armed Bandit
Transfer learning for extremely low resource languages is a challenging task as there is no large scale monolingual corpora for pre training or sufficient annotated data for fine tuning. We follow the work of MetaXL whic…
Meta-LearningNERTransfer LearningEvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing a…
Cross-Domain Few-Shotcross-domain few-shot learningFew-Shot LearningHyperparameter Optimization+1A Common Semantic Space for Monolingual and Cross-Lingual Meta-Embeddings
This paper presents a new technique for creating monolingual and cross-lingual meta-embeddings. Our method integrates multiple word embeddings created from complementary techniques, textual sources, knowledge bases and l…
Cross-Lingual TransferPOSPOS TaggingSemantic Textual Similarity+3MetaEnzyme: Meta Pan-Enzyme Learning for Task-Adaptive Redesign
Enzyme design plays a crucial role in both industrial production and biology. However, this field faces challenges due to the lack of comprehensive benchmarks and the complexity of enzyme design tasks, leading to a deart…
FLAT: Few-Shot Learning via Autoencoding Transformation Regularizers
One of the most significant challenges facing a few-shot learning task is the generalizability of the (meta-)model from the base to the novel categories. Most of existing few-shot learning models attempt to address this …
Data AugmentationFew-Shot LearningNovel Concepts