Predicting the Performance of Multilingual NLP Models
Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages. The languages that these models are evaluated on, however, are very few in number, and it is unlikely that evaluation datasets will cover all the languages that these models support. Potential solutions to the costly problem of dataset creation are to translate datasets to new languages or use template-filling based techniques for creation. This paper proposes an alternate solution for evaluating a model across languages which make use of the existing performance scores of the model on languages that a particular task has test sets for. We train a predictor on these performance scores and use this predictor to predict the model's performance in different evaluation settings. Our results show that our method is effective in filling the gaps in the evaluation for an existing set of languages, but might require additional improvements if we want it to generalize to unseen languages.
Code (0)
등록된 구현이 없습니다.
Tasks
Multilingual NLPMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration
Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabil…
PositionMultilingual and Multimodal Topic Modelling with Pretrained Embeddings
This paper presents M3L-Contrast -- a novel multimodal multilingual (M3L) neural topic model for comparable data that maps texts from multiple languages and images into a shared topic space. Our model is trained jointly …
Predicting Code-switching in Multilingual Communication for Immigrant Communities
The Dabblers at SemEval-2018 Task 2: Multilingual Emoji Prediction
The {``}Multilingual Emoji Prediction{''} task focuses on the ability of predicting the correspondent emoji for a certain tweet. In this paper, we investigate the relation between words and emojis. In order to do that, w…
BIG-bench Machine LearningRelationTask 2Unsupervised Translation Quality Estimation Exploiting Synthetic Data and Pre-trained Multilingual Encoder
Translation quality estimation (TQE) is the task of predicting translation quality without reference translations. Due to the enormous cost of creating training data for TQE, only a few translation directions can benefit…
SentenceTranslation