Low-resource Bilingual Dialect Lexicon Induction with Large Language Models
Bilingual word lexicons are crucial tools for multilingual natural language understanding and machine translation tasks, as they facilitate the mapping of words in one language to their synonyms in another language. To achieve this, numerous papers have explored bilingual lexicon induction (BLI) in high-resource scenarios, using a typical pipeline consisting of two unsupervised steps: bitext mining and word alignment, both of which rely on pre-trained large language models~(LLMs). In this paper, we present an analysis of the BLI pipeline for German and two of its dialects, Bavarian and Alemannic. This setup poses several unique challenges, including the scarcity of resources, the relatedness of the languages, and the lack of standardization in the orthography of dialects. To evaluate the BLI outputs, we analyze them with respect to word frequency and pairwise edit distance. Additionally, we release two evaluation datasets comprising 1,500 bilingual sentence pairs and 1,000 bilingual word pairs. They were manually judged for their semantic similarity for each Bavarian-German and Alemannic-German language pair.
Code (1)
Tasks
Bilingual Lexicon InductionMachine TranslationNatural Language UnderstandingSemantic SimilaritySemantic Textual SimilaritySentenceWord AlignmentSimilar Papers 제목 키워드 기반
Resource-Lean Lexicon Induction for German Dialects
Automatic induction of high-quality dictionaries is essential for building lexical resources, yet low-resource languages and dialects pose several challenges: limited access to annotators, high degree of spelling variati…
Information RetrievalBilingual Lexicon Induction across Orthographically-distinct Under-Resourced Dravidian Languages
Bilingual lexicons are a vital tool for under-resourced languages and recent state-of-the-art approaches to this leverage pretrained monolingual word embeddings using supervised or semi-supervised approaches. However, th…
Bilingual Lexicon InductionWord EmbeddingsBilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport
Bilingual lexicons form a critical component of various natural language processing applications, including unsupervised and semisupervised machine translation and crosslingual information retrieval. We improve bilingual…
Bilingual Lexicon InductionGraph MatchingInformation RetrievalMachine Translation+2Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport
Bilingual lexicons form a critical component of various NLP applications, including unsupervised and semisupervised machine translation and crosslingual information retrieval. In this work, we improve bilingual lexicon i…
Bilingual Lexicon InductionGraph MatchingInformation RetrievalMachine Translation+2ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting
Bilingual Lexicon Induction (BLI), where words are translated between two languages, is an important NLP task. While noticeable progress on BLI in rich resource languages using static word embeddings has been achieved. T…
Bilingual Lexicon InductionLanguage ModelingLanguage ModellingRe-Ranking+3