Lexical Normalization for Code-switched Data and its Effect on POS-tagging
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of manynatural language processing tasks on social media. Yet, using multiple languages in one utterance, also called code-switching (CS), is frequently overlooked by these normalization systems, despite its common use in social media. In this paper, we propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English (Id-En) and Turkish-German (Tr-De). For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models outperform Id-En state of the art and Tr-De monolingual models, and lead to 5.4% relative performance increase for POS tagging as compared to unnormalized input.
Code (0)
등록된 구현이 없습니다.
Tasks
Language IdentificationLexical NormalizationPOSPOS TaggingTranslationSimilar Papers 제목 키워드 기반
Lexical Normalization for Code-switched Data and its Effect on POS Tagging
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of many natural language processing tasks on social media. Yet, using multiple languages in one utte…
Lexical NormalizationPOSPOS TaggingTranslationAdapting Deep Learning for Sentiment Classification of Code-Switched Informal Short Text
Nowadays, an abundance of short text is being generated that uses nonstandard writing styles influenced by regional languages. Such informal and code-switched content are under-resourced in terms of labeled datasets and …
ClassificationGeneral ClassificationLexical NormalizationSentiment Analysis+2Generating Code-switched Text for Lexical Learning
Normalization and Back-Transliteration for Code-Switched Data
Code-switching is an omnipresent phenomenon in multilingual communities all around the world but remains a challenge for NLP systems due to the lack of proper data and processing techniques. Hindi-English code-switched t…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Part-Of-Speech Tagging+1Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing
Lexical normalization is the task of translating non-standard social media data to a standard form. Previous work has shown that this is beneficial for many downstream tasks in multiple languages. However, for Italian, t…
Dependency ParsingLexical Normalization