VarClass: An Open-source Language Identification Tool for Language Varieties
This paper presents VarClass, an open-source tool for language identification available both to be downloaded as well as through a graphical user-friendly interface. The main difference of VarClass in comparison to other state-of-the-art language identification tools is its focus on language varieties. General purpose language identification tools do not take language varieties into account and our work aims to fill this gap. VarClass currently contains language models for over 27 languages in which 10 of them are language varieties. We report an average performance of over 90.5{\%} accuracy in a challenging dataset. More language models will be included in the upcoming months.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalLanguage IdentificationMachine TranslationText CategorizationSimilar Papers 제목 키워드 기반
CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing
We present CAMeL Tools, a collection of open-source tools for Arabic natural language processing in Python. CAMeL Tools currently provides utilities for pre-processing, morphological modeling, Dialect Identification, Nam…
Arabic Sentiment AnalysisArabic Text DiacritizationDialect IdentificationMorphological Analysis+5GlotScript: A Resource and Tool for Low Resource Writing System Identification
We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than 7,000 languages. It is compiled by aggr…
Language ModelingLanguage ModellingMAPA Project: Ready-to-Go Open-Source Datasets and Deep Learning Technology to Remove Identifying Information from Text Documents
This paper presents the outcomes of the MAPA project, a set of annotated corpora for 24 languages of the European Union and an open-source customisable toolkit able to detect and substitute sensitive information in text …
De-identificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)A Software Toolkit for Pre-processing Sign Language Video Streams
We present the requirements, design guidelines, and the software architecture of an open-source toolkit dedicated to the pre-processing of sign language video material. The toolkit is a collection of functions and comman…
The Multilingual Anonymisation Toolkit for Public Administrations (MAPA) Project
We describe the MAPA project, funded under the Connecting Europe Facility programme, whose goal is the development of an open-source de-identification toolkit for all official European Union languages. It will be develop…
De-identification