CL-NERIL: A Cross-Lingual Model for NER in Indian Languages
Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a target Indian language. We employ a variant of the Teacher-Student model and optimize it jointly on the pseudo labels of the Teacher model and predictions on the generated weakly labeled data. We also present manually annotated test sets for three Indian languages: Hindi, Bengali, and Gujarati. We evaluate the performance of the proposed framework on the test sets of the three Indian languages. Empirical results show a minimum 10% performance improvement compared to the zero-shot transfer learning model on all languages. This indicates that weakly labeled data generated using the proposed annotation projection method in target Indian languages can complement well-annotated source language data to enhance performance. Our code is publicly available at https://github.com/aksh555/CL-NERIL
Code (1)
Tasks
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERTAGTransfer LearningWord AlignmentSimilar Papers 제목 키워드 기반
Everyday Speech in the Indian Subcontinent
India has 1369 languages of which 22 are official. About 13 different scripts are used to represent these languages. A Common Label Set (CLS) was developed based on phonetics to address the issue of large vocabulary of u…
Speech SynthesisEvaluating Inter-Bilingual Semantic Parsing for Indian Languages
Despite significant progress in Natural Language Generation for Indian languages (IndicNLP), there is a lack of datasets around complex structured tasks such as semantic parsing. One reason for this imminent gap is the c…
Semantic ParsingText GenerationTranslationAnalysing cross-lingual transfer in lemmatisation for Indian languages
Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. However, most of the prior work on this topic has focused on high resource languages. In this paper, …
Cross-Lingual TransferLemmatizationDuDe: Dual-Decoder Multilingual ASR for Indian Languages using Common Label Set
In a multilingual country like India, multilingual Automatic Speech Recognition (ASR) systems have much scope. Multilingual ASR systems exhibit many advantages like scalability, maintainability, and improved performance …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Decoderspeech-recognition+2IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages
In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c)…
Genre classificationMultiple-choiceMultiple Choice Question Answering (MCQA)named-entity-recognition+8