Discriminating between Similar Languages Using a Combination of Typed and Untyped Character N-grams and Words
This paper presents the cic{\_}ualg{'}s system that took part in the Discriminating between Similar Languages (DSL) shared task, held at the VarDial 2017 Workshop. This year{'}s task aims at identifying 14 languages across 6 language groups using a corpus of excerpts of journalistic texts. Two classification approaches were compared: a single-step (all languages) approach and a two-step (language group and then languages within the group) approach. Features exploited include lexical features (unigrams of words) and character n-grams. Besides traditional (untyped) character n-grams, we introduce typed character n-grams in the DSL task. Experiments were carried out with different feature representation methods (binary and raw term frequency), frequency threshold values, and machine-learning algorithms {--} Support Vector Machines (SVM) and Multinomial Naive Bayes (MNB). Our best run in the DSL task achieved 91.46{\%} accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationInformation RetrievalMachine TranslationSimilar Papers 제목 키워드 기반
Discriminating Similar Languages: Evaluations and Explorations
We present an analysis of the performance of machine learning classifiers on discriminating between similar languages and language varieties. We carried out a number of experiments using the results of the two editions o…
BIG-bench Machine LearningImproving Language Identification for Multilingual Speakers
Spoken language identification (LID) technologies have improved in recent years from discriminating largely distinct languages to discriminating highly similar languages or even dialects of the same language. One aspect …
Language IdentificationSpoken language identificationDiscriminating Between Similar Nordic Languages
Automatic language identification is a challenging problem. Discriminating between closely related languages is especially difficult. This paper presents a machine learning approach for automatic language identification …
BIG-bench Machine LearningLanguage IdentificationFindings of the VarDial Evaluation Campaign 2023
This report presents the results of the shared tasks organized as part of the VarDial Evaluation Campaign 2023. The campaign is part of the tenth workshop on Natural Language Processing (NLP) for Similar Languages, Varie…
Intent DetectionASIREM Participation at the Discriminating Similar Languages Shared Task 2016
This paper presents the system built by ASIREM team for the Discriminating between Similar Languages (DSL) Shared task 2016. It describes the system which uses character-based and word-based n-grams separately. ASIREM pa…
Dialect IdentificationLanguage IdentificationTask 2