LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl
The web data contains immense amount of data, hundreds of billion words are waiting to be extracted and used for language research. In this work we introduce our tool LanguageCrawl which allows NLP researchers to easily construct web-scale corpus from Common Crawl Archive: a petabyte scale, open repository of web crawl information. Three use-cases are presented: filtering Polish websites, building an N-gram corpora and training continuous skip-gram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and N-gram ranks. Special effort has been put on high computing efficiency, by applying highly concurrent multitasking. We make our tool publicly available to enrich NLP resources. We strongly believe that our work will help to facilitate NLP research, especially in under-resourced languages, where the lack of appropriately sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
HAMMR: HierArchical MultiModal React agents for generic VQA
Combining Large Language Models (LLMs) with external specialized tools (LLMs+tools) is a recent paradigm to solve multimodal tasks such as Visual Question Answering (VQA). While this approach was demonstrated to work wel…
Optical Character Recognition (OCR)Question AnsweringSpatial ReasoningVisual Question Answering+1Building a Multi-domain Neural Machine Translation Model using Knowledge Distillation
Lack of specialized data makes building a multi-domain neural machine translation tool challenging. Although emerging literature dealing with low resource languages starts to show promising results, most state-of-the-art…
Domain AdaptationKnowledge DistillationMachine TranslationTranslationExtensions of Generic DOL for Generic Ontology Design Patterns
Generic ontologies were introduced as an extension (Generic DOL) of the Distributed Ontology, Modeling and Specification Language, DOL, with the aim to provide a language for Generic Ontology Design Patterns. In this pap…
Clash of the models: Comparing performance of BERT-based variants for generic news frame detection
Framing continues to remain one of the most extensively applied theories in political communication. Developments in computation, particularly with the introduction of transformer architecture and more so with large lang…
Learning Efficient Unsupervised Satellite Image-based Building Damage Detection
Existing Building Damage Detection (BDD) methods always require labour-intensive pixel-level annotations of buildings and their conditions, hence largely limiting their applications. In this paper, we investigate a chall…
Building Damage AssessmentDamaged Building DetectionDisaster ResponseExtracting Buildings In Remote Sensing Images+1