Average Is Not Enough: Caveats of Multilingual Evaluation
This position paper discusses the problem of multilingual evaluation. Using simple statistics, such as average language performance, might inject linguistic biases in favor of dominant language families into evaluation methodology. We argue that a qualitative analysis informed by comparative linguistics is needed for multilingual results to detect this kind of bias. We show in our case study that results in published works can indeed be linguistically biased and we demonstrate that visualization based on URIEL typological database can detect it.
Code (1)
Tasks
PositionSimilar Papers 제목 키워드 기반
Caveats of Measuring Semantic Change of Cognates and Borrowings using Multilingual Word Embeddings
Cognates and borrowings carry different aspects of etymological evolution. In this work, we study semantic change of such items using multilingual word embeddings, both static and contextualised. We underline caveats ide…
Multilingual Word EmbeddingsWord EmbeddingsEl Departamento de Nosotros: How Machine Translated Corpora Affects Language Models in MRC Tasks
Pre-training large-scale language models (LMs) requires huge amounts of text corpora. LMs for English enjoy ever growing corpora of diverse language resources. However, less resourced languages and their mono- and multil…
Machine TranslationQuestion AnsweringTransfer LearningTranslationLocal Structure Matters Most in Most Languages
Many recent perturbation studies have found unintuitive results on what does and does not matter when performing Natural Language Understanding (NLU) tasks in English. Coding properties, such as the order of words, can o…
Natural Language UnderstandingPhoneme transcription of endangered languages: an evaluation of recent ASR architectures in the single speaker scenario
Transcription is often reported as the bottleneck in endangered language documentation, requiring large efforts from scarce speakers and transcribers. In general, automatic speech recognition (ASR) can be accurate enough…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Phoneme Recognitionspeech-recognition+1An Efficient Multilingual Language Model Compression through Vocabulary Trimming
Multilingual language model (LM) have become a powerful tool in NLP especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabula…
Language ModelingLanguage ModellingModel Compression