Lexical Normalization for Code-switched Data and its Effect on POS Tagging
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of many natural language processing tasks on social media. Yet, using multiple languages in one utterance, also called code-switching (CS), is frequently overlooked by these normalization systems, despite its common use in social media. In this paper, we propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English and Turkish-German. For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models significantly outperform monolingual ones, and lead to 5.4{\%} relative performance increase for POS tagging as compared to unnormalized input.
Code (1)
Tasks
Lexical NormalizationPOSPOS TaggingTranslationSimilar Papers 제목 키워드 기반
Lexical Normalization for Code-switched Data and its Effect on POS-tagging
Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of manynatural language processing tasks on social media. Yet, using multiple languages in one utter…
Language IdentificationLexical NormalizationPOSPOS Tagging+1Adapting Deep Learning for Sentiment Classification of Code-Switched Informal Short Text
Nowadays, an abundance of short text is being generated that uses nonstandard writing styles influenced by regional languages. Such informal and code-switched content are under-resourced in terms of labeled datasets and …
ClassificationGeneral ClassificationLexical NormalizationSentiment Analysis+2Generating Code-switched Text for Lexical Learning
Normalization and Back-Transliteration for Code-Switched Data
Code-switching is an omnipresent phenomenon in multilingual communities all around the world but remains a challenge for NLP systems due to the lack of proper data and processing techniques. Hindi-English code-switched t…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Part-Of-Speech Tagging+1Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing
Lexical normalization is the task of translating non-standard social media data to a standard form. Previous work has shown that this is beneficial for many downstream tasks in multiple languages. However, for Italian, t…
Dependency ParsingLexical Normalization