Extracting Weighted Language Lexicons from Wikipedia
Language models are used in applications as diverse as speech recognition, optical character recognition and information retrieval. They are used to predict word appearance, and to weight the importance of words in these applications. One basic element of language models is the list of words in a language. Another is the unigram frequency of each word. But this basic information is not available for most languages in the world. Since the multilingual Wikipedia project encourages the production of encyclopedic-like articles in many world languages, we can find there an ever-growing source of text from which to extract these two language modelling elements: word list and frequency. Here we present a simple technique for converting this Wikipedia text into lexicons of weighted unigrams for the more than 280 languages present currently present in Wikipedia. The lexicons produced, and the source code for producing them in a Linux-based system are here made available for free on the Web.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesInformation RetrievalLanguage ModellingOptical Character RecognitionOptical Character Recognition (OCR)Retrievalspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Mind Your POV: Convergence of Articles and Editors Towards Wikipedia's Neutrality Norm
Wikipedia has a strong norm of writing in a 'neutral point of view' (NPOV). Articles that violate this norm are tagged, and editors are encouraged to make corrections. But the impact of this tagging system has not been q…
ArticlesTime SeriesTime Series AnalysisPolarity Lexicon Building: to what Extent Is the Manual Effort Worth?
Polarity lexicons are a basic resource for analyzing the sentiments and opinions expressed in texts in an automated way. This paper explores three methods to construct polarity lexicons: translating existing lexicons fro…
A Parallel English - Serbian - Bulgarian - Macedonian Lexicon of Named Entities
This paper describes the creation of a parallel multilingual lexicon of named entities from English to three South Slavic languages: Serbian, Bulgarian and Macedonian, with Wikipedia as a source. The basics of the propos…
MiscellaneousProcessing South Asian Languages Written in the Latin Script: the Dakshina Dataset
This paper describes the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. The dataset includes, for each language: 1) native script Wikipedia text; 2)…
Language ModelingLanguage ModellingSentenceTransliterationInferring Gender: A Scalable Methodology for Gender Detection with Online Lexical Databases
This paper presents a new method for automatic detection of gendered terms in large-scale language datasets. Currently, the evaluation of gender bias in natural language processing relies on the use of manually compiled …