Comparing Fifty Natural Languages and Twelve Genetic Languages Using Word Embedding Language Divergence (WELD) as a Quantitative Measure of Language Distance
We introduce a new measure of distance between languages based on word embedding, called word embedding language divergence (WELD). WELD is defined as divergence between unified similarity distribution of words between languages. Using such a measure, we perform language comparison for fifty natural languages and twelve genetic languages. Our natural language dataset is a collection of sentence-aligned parallel corpora from bible translations for fifty languages spanning a variety of language families. Although we use parallel corpora, which guarantees having the same content in all languages, interestingly in many cases languages within the same family cluster together. In addition to natural languages, we perform language comparison for the coding regions in the genomes of 12 different organisms (4 plants, 6 animals, and two human subjects). Our result confirms a significant high-level difference in the genetic language model of humans/animals versus plants. The proposed method is a step toward defining a quantitative measure of similarity between languages, with applications in languages classification, genre identification, dialect identification, and evaluation of translations.
Code (0)
등록된 구현이 없습니다.
Tasks
Dialect IdentificationLanguage ModelingLanguage ModellingSentenceSimilar Papers 제목 키워드 기반
Challenge Dataset of Cognates and False Friend Pairs from Indian Languages
Cognates are present in multiple variants of the same text across different languages (e.g., "hund" in German and "hound" in English language mean "dog"). They pose a challenge to various Natural Language Processing (NLP…
Information RetrievalMachine TranslationRetrievalTranslationHuman genetic admixture through the lens of population genomics
Over the last fifty years, geneticists have made great strides in understanding how our species' evolutionary history gave rise to current patterns of human genetic diversity classically summarized by Lewontin in his 197…
DiversityMeasuring the Similarity of Grammatical Gender Systems by Comparing Partitions
A grammatical gender system divides a lexicon into a small number of relatively fixed grammatical categories. How similar are these gender systems across languages? To quantify the similarity, we define gender systems ex…
Community DetectionWhat do character-level models learn about morphology? The case of dependency parsing
When parsing morphologically-rich languages with neural models, it is beneficial to model input at the character level, and it has been claimed that this is because character-level models learn morphology. We test these …
Dependency ParsingMorphological AnalysisHarnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages
Cognates are variants of the same lexical form across different languages; for example 'fonema' in Spanish and 'phoneme' in English are cognates, both of which mean 'a unit of sound'. The task of automatic detection of c…
Cross-Lingual Information RetrievalCross-Lingual Word EmbeddingsInformation RetrievalMachine Translation+4