An Unsupervised Morphological Criterion for Discriminating Similar Languages
In this study conducted on the occasion of the Discriminating between Similar Languages shared task, I introduce an additional decision factor focusing on the token and subtoken level. The motivation behind this submission is to test whether a morphologically-informed criterion can add linguistically relevant information to global categorization and thus improve performance. The contributions of this paper are (1) a description of the unsupervised, low-resource method; (2) an evaluation and analysis of its raw performance; and (3) an assessment of its impact within a model comprising common indicators used in language identification. I present and discuss the systems used in the task A, a 12-way language identification task comprising varieties of five main language groups. Additionally I introduce a new off-the-shelf Naive Bayes classifier using a contrastive word and subword n-gram model ({``}Bayesline{''}) which outperforms the best submissions.
Code (0)
등록된 구현이 없습니다.
Tasks
Language IdentificationText CategorizationSimilar Papers 제목 키워드 기반
Discriminating Similar Languages: Evaluations and Explorations
We present an analysis of the performance of machine learning classifiers on discriminating between similar languages and language varieties. We carried out a number of experiments using the results of the two editions o…
BIG-bench Machine LearningUnsupervised Morphological Expansion of Small Datasets for Improving Word Embeddings
We present a language independent, unsupervised method for building word embeddings using morphological expansion of text. Our model handles the problem of data sparsity and yields improved word embeddings by relying on …
Word EmbeddingsWord SimilarityUnsupervised Morphological Segmentation for Low-Resource Polysynthetic Languages
Polysynthetic languages pose a challenge for morphological analysis due to the root-morpheme complexity and to the word class {``}squish{''}. In addition, many of these polysynthetic languages are low-resource. We propos…
Morphological AnalysisAutomatically Tailoring Unsupervised Morphological Segmentation to the Language
Morphological segmentation is beneficial for several natural language processing tasks dealing with large vocabularies. Unsupervised methods for morphological segmentation are essential for handling a diverse set of lang…
Machine TranslationSegmentationSpeech RecognitionUnsupervised Inflection Generation Using Neural Language Modeling
The use of Deep Neural Network architectures for Language Modeling has recently seen a tremendous increase in interest in the field of NLP with the advent of transfer learning and the shift in focus from rule-based and p…
Language ModelingLanguage ModellingQuestion AnsweringTransfer Learning