Word-level Language Identification Using Subword Embeddings for Code-mixed Bangla-English Social Media Data
This paper reports work on building a word-level language identification (LID) model for code-mixed Bangla-English social media data using subword embeddings, with an ultimate goal of using this LID module as the first step in a modular part-of-speech (POS) tagger in future research. This work reports preliminary results of a word-level LID model that uses a single bidirectional LSTM with subword embeddings trained on very limited code-mixed resources. At the time of writing, there are no previous reported results available in which subword embeddings are used for language identification with the Bangla-English code-mixed language pair. As part of the current work, a labeled resource for word-level language identification is also presented, by correcting 85.7% of labels from the 2016 ICON Whatsapp Bangla-English dataset. The trained model was evaluated on a test set of 4,015 tokens compiled from the 2015 and 2016 ICON datasets, and achieved a test accuracy of 93.61%.
Code (1)
Tasks
Language IdentificationPOSSimilar Papers 제목 키워드 기반
Character-based Neural Networks for Sentence Pair Modeling
Sentence pair modeling is critical for many NLP tasks, such as paraphrase identification, semantic textual similarity, and natural language inference. Most state-of-the-art neural models for these tasks rely on pretraine…
Language ModelingLanguage ModellingNatural Language InferenceParaphrase Identification+4Subword-level Composition Functions for Learning Word Embeddings
Subword-level information is crucial for capturing the meaning and morphology of words, especially for out-of-vocabulary entries. We propose CNN- and RNN-based subword-level composition functions for learning word embedd…
Learning Word EmbeddingsWord EmbeddingsPBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding
We look into the task of \emph{generalizing} word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, \emph{without} extra c…
POSPOS TaggingWord EmbeddingsWord SimilarityNeural Named Entity Recognition from Subword Units
Named entity recognition (NER) is a vital task in spoken language understanding, which aims to identify mentions of named entities in text e.g., from transcribed speech. Existing neural models for NER rely mostly on dedi…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+1Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models
Human understanding of language is robust to different word choices as far as they represent similar semantic concepts. To what extent does our human intuition transfer to language models, which represent all subwords as…