FuLG: 150B Romanian Corpus for Language Model Pretraining
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingmodelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Neural Grammatical Error Correction for Romanian
Resources for Grammatical Error Correction (GEC) in non-English languages are scarce, while available spellcheckers in these languages are mostly limited to simple corrections and rules. In this paper we introduce a firs…
Grammatical Error CorrectionLLMic: Romanian Foundation Language Model
Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks with commercial models leading the way. While open models usually operate at a smaller scale, they maintain c…
Language ModelingLanguage ModellingmodelTranslationTF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction
Recent advances in synthetic data generation have shown that compact language models can be trained effectively when the underlying corpus is structurally controlled and linguistically coherent. However, for morphologica…
Synthetic Data GenerationKnowledge DistillationImproving Romanian LLM Pretraining Data using Diversity and Quality Filtering
Large Language Models (LLMs) have recently exploded in popularity, often matching or outperforming human abilities on many tasks. One of the key factors in training LLMs is the availability and curation of high-quality d…
Adapting the TTL Romanian POS Tagger to the Biomedical Domain
This paper presents the adaptation of the Hidden Markov Models-based TTL part-of-speech tagger to the biomedical domain. TTL is a text processing platform that performs sentence splitting, tokenization, POS tagging, chun…
ChunkingDomain AdaptationLemmatizationnamed-entity-recognition+8