LanideNN: Multilingual Language Identification on Character Window
In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three languages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages. Our method is based on Bidirectional Recurrent Neural Networks and it performs well in monolingual and multilingual language identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short documents and across domains, so it is ideal for off-the-shelf use without preparation of training data.
Code (1)
Tasks
Language IdentificationSimilar Papers 제목 키워드 기반
CoSwID, a Code Switching Identification Method Suitable for Under-Resourced Languages
We propose a method for identifying monolingual textual segments in multilingual documents. It requires only a minimal number of linguistic resources – word lists and monolingual corpora – and can therefore be adapted to…
Language IdentificationLarge-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning
In this paper, we report a large-scale end-to-end language-independent multilingual model for joint automatic speech recognition (ASR) and language identification (LID). This model adopts hybrid CTC/attention architectur…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language IdentificationMulti-Task Learning+2MultiVitaminBooster at PARSEME Shared Task 2020: Combining Window- and Dependency-Based Features with Multilingual Contextualised Word Embeddings for VMWE Detection
In this paper, we present MultiVitaminBooster, a system implemented for the PARSEME shared task on semi-supervised identification of verbal multiword expressions - edition 1.2. For our approach, we interpret detecting ve…
Language ModelingLanguage Modellingtoken-classificationToken Classification+1Incorporating Dialectal Variability for Socially Equitable Language Identification
Language identification (LID) is a critical first step for processing multilingual text. Yet most LID systems are not designed to handle the linguistic diversity of global platforms like Twitter, where local dialects and…
DiversityLanguage IdentificationLanguage Lexicons for Hindi-English Multilingual Text Processing
Language Identification in textual documents is the process of automatically detecting the language contained in a document based on its content. The present Language Identification techniques presume that a document con…
Language Identification