Open-Set Language Identification
We present the first open-set language identification experiments using one-class classification. We first highlight the shortcomings of traditional feature extraction methods and propose a hashing-based feature vectorization approach as a solution. Using a dataset of 10 languages from different writing systems, we train a One- Class Support Vector Machine using only a monolingual corpus for each language. Each model is evaluated against a test set of data from all 10 languages and we achieve an average F-score of 0.99, highlighting the effectiveness of this approach for open-set language identification.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationLanguage IdentificationOne-Class ClassificationSimilar Papers 제목 키워드 기반
Robust Open-Set Spoken Language Identification and the CU MultiLang Dataset
Most state-of-the-art spoken language identification models are closed-set; in other words, they can only output a language label from the set of classes they were trained on. Open-set spoken language identification syst…
Language IdentificationSpoken language identificationVarClass: An Open-source Language Identification Tool for Language Varieties
This paper presents VarClass, an open-source tool for language identification available both to be downloaded as well as through a graphical user-friendly interface. The main difference of VarClass in comparison to other…
Information RetrievalLanguage IdentificationMachine TranslationText CategorizationModernizing Open-Set Speech Language Identification
While most modern speech Language Identification methods are closed-set, we want to see if they can be modified and adapted for the open-set problem. When switching to the open-set problem, the solution gains the ability…
Language IdentificationSpeech Language IdentificationA reproduction of Apple's bi-directional LSTM models for language identification in short strings
Language Identification is the task of identifying a document's language. For applications like automatic spell checker selection, language identification must use very short strings such as text message fragments. In th…
Language IdentificationOpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report
Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to…
Language Identification