AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indicnlp_corpus.
Code (2)
Tasks
Word EmbeddingsSimilar Papers 제목 키워드 기반
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages
In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c)…
Genre classificationMultiple-choiceMultiple Choice Question Answering (MCQA)named-entity-recognition+8Building a Monolingual Parallel Corpus for Text Simplification Using Sentence Similarity Based on Alignment between Word Embeddings
Methods for text simplification using the framework of statistical machine translation have been extensively studied in recent years. However, building the monolingual parallel corpus necessary for training the model req…
Machine TranslationSentenceSentence SimilarityText Simplification+2Estimating related words computationally using language model from the Mahabharata - an Indian epic
'Mahabharata' is the most popular among many Indian pieces of literature referred to in many domains for completely different purposes. This text itself is having various dimension and aspects which is useful for the hum…
Language ModelingLanguage ModellingSentenceMonolingual and Parallel Corpora for Kangri Low Resource Language
In this paper we present the dataset of Himachali low resource endangered language, Kangri (ISO 639-3xnr) listed in the United Nations Educational, Scientific and Cultural Organization (UNESCO). The compilation of kangri…
Machine TranslationNMTTranslationWord EmbeddingsNoisy Parallel Corpus Filtering through Projected Word Embeddings
We present a very simple method for parallel text cleaning of low-resource languages, based on projection of word embeddings trained on large monolingual corpora in high-resource languages. In spite of its simplicity, we…
Machine TranslationTranslationWord Embeddings