paper-with-me

홈 › Papers

Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

2026-09-07 · Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, Daniel Gomes, João Magalhães arxiv

Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.

📄 PDF Abstract BibTeX arXiv:2609.07699

Code (0)

등록된 구현이 없습니다.

Tasks

Language Identification

Similar Papers 제목 키워드 기반

DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models

2024-10-08 · Ranchi Zhao, Zhen Leng Thai, Yifan Zhang, Shengding Hu 외

The performance of Large Language Models (LLMs) is substantially influenced by the pretraining corpus, which consists of vast quantities of unsupervised data processed by the models. Despite its critical role in model pe…

Language ModelingLanguage ModellingLarge Language ModelSmall Language Model

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only

2023-09-26 · NeurIPS 2023 11

Large language models are commonly trained on a mixture of filtered web data and curated ``high-quality'' corpora, such as social media conversations, books, or technical papers. This curation process is believed to be n…

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

2023-06-01 · Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru 외

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be neces…

Zero-shot Generalization

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

2024-09-23 · Clément Christophe, Tathagata Raha, Svetlana Maslenkova, Muhammad Umar Salman 외

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuou…

Prompt Engineering

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

2024-02-29 · Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang 외

This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for langua…