Modeling Code-Switch Languages Using Bilingual Parallel Corpus
Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual sequential dependency. The attention mechanism learns the bilingual context from a parallel corpus. BALM achieves state-of-the-art performance on the SEAME code-switch database by reducing the perplexity of 20.5{\%} over the best-reported result. We also apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilingual Lexicon InductionLanguage ModelingLanguage ModellingTranslationSimilar Papers 제목 키워드 기반
Code-switching Language Modeling With Bilingual Word Embeddings: A Case Study for Egyptian Arabic-English
Code-switching (CS) is a widespread phenomenon among bilingual and multilingual societies. The lack of CS resources hinders the performance of many NLP tasks. In this work, we explore the potential use of bilingual word …
Language ModelingLanguage ModellingWord EmbeddingsStreaming Bilingual End-to-End ASR model using Attention over Multiple Softmax
Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availabil…
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining
Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details o…
Dual Language Models for Code Switched Speech Recognition
In this work, we present a simple and elegant approach to language modeling for bilingual code-switched text. Since code-switching is a blend of two or more different languages, a standard bilingual language model can be…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+2Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning
Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). Howe…
Cross-Lingual Transfer