Code-Switching Language Modeling using Syntax-Aware Multi-Task Learning
Lack of text data has been the major issue on code-switching language modeling. In this paper, we introduce multi-task learning based language model which shares syntax representation of languages to leverage linguistic information and tackle the low resource data issue. Our model jointly learns both language modeling and Part-of-Speech tagging on code-switched utterances. In this way, the model is able to identify the location of code-switching points and improves the prediction of next word. Our approach outperforms standard LSTM based language model, with an improvement of 9.7% and 7.4% in perplexity on SEAME Phase I and Phase II dataset respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMulti-Task LearningPart-Of-Speech TaggingSyntax RepresentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Language Modeling for Code-Switched Data: Challenges and Approaches
Lately, the problem of code-switching has gained a lot of attention and has emerged as an active area of research. In bilingual communities, the speakers commonly embed the words and phrases of a non-native language into…
Language ModelingLanguage ModellingPOSEntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching
Accurate alignment between languages is fundamental for improving cross-lingual pre-trained language models (XLMs). Motivated by the natural phenomenon of code-switching (CS) in multilingual speakers, CS has been used as…
Cross-Lingual TransferData AugmentationRetrievalSentence+2Building a User-Generated Content North-African Arabizi Treebank: Tackling Hell
We introduce the first treebank for a romanized user-generated content variety of Algerian, a North-African Arabic dialect known for its frequent usage of code-switching. Made of 1500 sentences, fully annotated in morpho…
Dependency ParsingPOSPOS TaggingSentence+1Text Modeling with Syntax-Aware Variational Autoencoders
Syntactic information contains structures and rules about how text sentences are arranged. Incorporating syntax into text modeling methods can potentially benefit both representation learning and generation. Variational …
Representation LearningContinual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network
In this paper, we study how to continually pre-train language models for improving the understanding of math problems. Specifically, we focus on solving a fundamental challenge in modeling math problems, how to fuse the …
Math