A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Base
In the pursuit of supporting more languages around the world, tools that characterize properties of languages play a key role in expanding the existing multilingual NLP research. In this study, we focus on a widely used typological knowledge base, URIEL, which aggregates linguistic information into numeric vectors. Specifically, we delve into the soundness and reproducibility of the approach taken by URIEL in quantifying language similarity. Our analysis reveals URIEL's ambiguity in calculating language distances and in handling missing values. Moreover, we find that URIEL does not provide any information about typological features for 31\% of the languages it represents, undermining the reliabilility of the database, particularly on low-resource languages. Our literature review suggests URIEL and lang2vec are used in papers on diverse NLP tasks, which motivates us to rigorously verify the database as the effectiveness of these works depends on the reliability of the information the tool provides.
Code (0)
등록된 구현이 없습니다.
Tasks
Missing ValuesMultilingual NLPMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art
This registered report introduces the largest, and for the first time, reproducible experimental survey on biomedical sentence similarity with the following aims: (1) to elucidate the state of the art of the problem; (2)…
Articlesnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3Quantifying perturbation impacts for large language models
We consider the problem of quantifying how an input perturbation impacts the outputs of large language models (LLMs), a fundamental task for model reliability and post-hoc interpretability. A key obstacle in this domain …
Semantic SimilaritySemantic Textual SimilarityQuantifying Reproducibility in NLP and ML
Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so far emerged. The assumption has been that…
Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications
Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can c…
Deep LearningGPUSensitivityFoundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness
Large Language Models (LLMs) encode substantial factual knowledge, yet measuring and systematizing this knowledge remains challenging. Converting it into structured format, for example through recursive extraction approa…
Semantic Similarity