Automatic Detection and Language Identification of Multilingual Documents
Language identification is the task of automatically detecting the language(s) present in a document based on the content of the document. In this work, we address the problem of detecting documents that contain text from more than one language (multilingual documents). We introduce a method that is able to detect that a document is multilingual, identify the languages present, and estimate their relative proportions. We demonstrate the effectiveness of our method over synthetic data, as well as real-world multilingual documents collected from the web.
Code (0)
등록된 구현이 없습니다.
Tasks
Language IdentificationMachine TranslationSimilar Papers 제목 키워드 기반
LanideNN: Multilingual Language Identification on Character Window
In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is w…
Language IdentificationHandwritten Script Identification from Text Lines
In a multilingual country like India where 12 different official scripts are in use, automatic identification of handwritten script facilitates many important applications such as automatic transcription of multilingual …
Optical Character RecognitionOptical Character Recognition (OCR)CoSwID, a Code Switching Identification Method Suitable for Under-Resourced Languages
We propose a method for identifying monolingual textual segments in multilingual documents. It requires only a minimal number of linguistic resources – word lists and monolingual corpora – and can therefore be adapted to…
Language IdentificationLanguage Lexicons for Hindi-English Multilingual Text Processing
Language Identification in textual documents is the process of automatically detecting the language contained in a document based on its content. The present Language Identification techniques presume that a document con…
Language IdentificationIdentification of Languages in Algerian Arabic Multilingual Documents
This paper presents a language identification system designed to detect the language of each word, in its context, in a multilingual documents as generated in social media by bilingual/multilingual communities, in our ca…
ChunkingGeneral ClassificationLanguage Identification