Generalization Measures for Zero-Shot Cross-Lingual Transfer
A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language model evaluation tasks lack information metrics about model generalization and their applicability in a new setting is measured using task and language-specific downstream performance, which is often lacking in many languages and tasks. In this paper, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models in cross-lingual zero-shot settings. In addition to traditional measures such as variance in parameters after training and distance from initialization, we also measure the effectiveness of sharpness in loss landscape in capturing the success in cross-lingual transfer and propose a novel and stable algorithm to reliably compute the sharpness of a model optimum that correlates to generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferLanguage Model EvaluationLanguage ModelingLanguage ModellingZero-Shot Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zero-shot Cross-lingual Transfer is Under-specified Optimization
Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that this high variance results fr…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferDiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer
Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low-resource languages. In this work, we envision languages as domains for improving zero…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferOn Zero-shot Cross-lingual Transfer of Multilingual Neural Machine Translation
Transferring representations from large-scale supervised tasks to downstream tasks have shown outstanding results in Machine Learning in both Computer Vision and natural language processing (NLP). One particular example …
Cross-Lingual TransferMachine TranslationNatural Language InferenceNMT+4Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection
We study the selection of transfer languages for automatic abusive language detection. Instead of preparing a dataset for every language, we demonstrate the effectiveness of cross-lingual transfer learning for zero-shot …
Abusive LanguageCross-Lingual TransferTransfer LearningmPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible …
Cross-Lingual TransferOpen-Ended Question AnsweringZero-Shot Cross-Lingual Transfer