Vanilla Classifiers for Distinguishing between Similar Languages

2016-12-01 · WS 2016 12 · Sergiu Nisioi, Alina Maria Ciobanu, Liviu P. Dinu

In this paper we describe the submission of the UniBuc-NLP team for the Discriminating between Similar Languages Shared Task, DSL 2016. We present and analyze the results we obtained in the closed track of sub-task 1 (Similar languages and language varieties) and sub-task 2 (Arabic dialects). For sub-task 1 we used a logistic regression classifier with tf-idf feature weighting and for sub-task 2 a character-based string kernel with an SVM classifier. Our results show that good accuracy scores can be obtained with limited feature and model engineering. While certain limitations are to be acknowledged, our approach worked surprisingly well for out-of-domain, social media data, with 0.898 accuracy (3rd place) for dataset B1 and 0.838 accuracy (4th place) for dataset B2.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Information Retrieval Language Identification Question Answering regressionTask 2

Methods 이 논문이 사용한 방법론

Logistic Regression Logistic Regression, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression,…

SVM A Support Vector Machine, or SVM, is a non-parametric supervised learning model. For non-linear classification and regression, they utilise the kernel trick to map inputs…

Applying the Cognitive Machine Translation Evaluation Approach to Arabic

2016-05-01 · LREC 2016 5 · Irina Temnikova, Wajdi Zaghouani, Stephan Vogel, Nizar Habash

The goal of the cognitive machine translation (MT) evaluation approach is to build classifiers which assign post-editing effort scores to new texts. The approach helps estimate fair compensation for post-editors in the t…

Machine Translation Translation

Discriminating Similar Languages: Evaluations and Explorations

2016-09-30 · LREC 2016 5 · Cyril Goutte, Serge Léger, Shervin Malmasi, Marcos Zampieri

We present an analysis of the performance of machine learning classifiers on discriminating between similar languages and language varieties. We carried out a number of experiments using the results of the two editions o…

BIG-bench Machine Learning

Character-level NMT and language similarity

2023-08-08 · Josef Jon, Ondřej Bojar

We explore the effectiveness of character-level neural machine translation using Transformer architecture for various levels of language similarity and size of the training dataset on translation between Czech and Croati…

Machine Translation NMT Segmentation Translation

Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks

2016-12-01 · WS 2016 12 · Chinnappa Guggilla

In this paper, we describe a system (CGLI) for discriminating similar languages, varieties and dialects using convolutional neural networks (CNNs) and long short-term memory (LSTM) neural networks. We have participated i…

Dialect Identification Information Retrieval Language Identification Machine Translation+2

Bilingual Word Level Language Identification for Omotic Languages

2025-09-05 · Mesay Gemeda Yigezu, Girma Yohannis Bade, Atnafu Lambebo Tonja, Olga Kolesnikova 외 arxiv

Language identification is the task of determining the languages for a given text. In many real world scenarios, text may contain more than one language, particularly in multilingual communities. Bilingual Language Ident…

Language Identification