All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media
In this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman correlation coefficient values, our methods perform more than two times better (nearly 0.62) in predicting the borrowing likeliness compared to the best performing baseline (nearly 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88 percent of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems.
Code (0)
등록된 구현이 없습니다.
Tasks
AllLanguage IdentificationTAGSimilar Papers 제목 키워드 기반
An Ensemble Approach for Aggression Identification in English and Hindi Text
This paper describes our system submitted in the shared task at COLING 2018 TRAC-1: Aggression Identification. The objective of this task was to predict online aggression spread through online textual post or comment. Th…
Aggression IdentificationLanguage Identification and Analysis of Code-Switched Social Media Text
In this paper, we detail our work on comparing different word-level language identification systems for code-switched Hindi-English data and a standard Spanish-English dataset. In this regard, we build a new code-switche…
Language IdentificationMachine TranslationMHE: Code-Mixed Corpora for Similar Language Identification
This paper introduces a new Magahi-Hindi-English (MHE) code-mixed data-set for similar language identification (SMLID), where Magahi is a less-resourced minority language. This corpus provides a language id at two levels…
Language IdentificationSentenceAggression Identification in English, Hindi and Bangla Text using BERT, RoBERTa and SVM
This paper presents the results of the classifiers we developed for the shared tasks in aggression identification and misogynistic aggression identification. These two shared tasks were held as part of the second worksho…
Aggression IdentificationMisogynistic Aggression IdentificationAll that is English may be Hindi: Enhancing language identification through automatic ranking of the likeliness of word borrowing in social media
n this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman{'}s correlation values, our methods perform more t…
AllLanguage IdentificationTAG