paper-with-me

홈 › Papers

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

2024-02-29 · Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, JIA YU, Chaobin Zhang, Zhenxiang Li, Pei Chu, Yuan Qu, Jin Shi, Lindong Lu, Runyu Peng, Zhiyuan Zeng, Huanze Tang, Zhikai Lei, Jiawei Hong, Keyu Chen, Zhaoye Fei, Ruiliang Xu, Wei Li, Zhongying Tu, Lin Dahua, Yu Qiao, Hang Yan, Conghui He

This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks.

📄 PDF Abstract BibTeX arXiv:2402.19282

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages

2025-01-24 · JIA YU, Fei Yuan, Rui Min, Jing Yu 외

This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve t…

Diversity

LongWanjuan: Towards Systematic Measurement for Long Text Quality

2024-02-21 · Kai Lv, Xiaoran Liu, Qipeng Guo, Hang Yan 외

The quality of training data are crucial for enhancing the long-text capabilities of foundation models. Despite existing efforts to refine data quality through heuristic rules and evaluations based on data diversity and …

DiversityLanguage Modelling

WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models

2023-08-21 · Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu 외

The rise in popularity of ChatGPT and GPT-4 has significantly accelerated the development of large models, leading to the creation of numerous impressive large language models(LLMs) and multimodal large language models (…

CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models

2024-10-24 · Liangdong Wang, Bo-Wen Zhang, ChengWei Wu, Hanyu Zhao 외

We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a high-quality 500GB subset of the Chinese Corpora Internet 3.0 (CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a novel two-s…

ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information

2024-11-29 · Wanyue Zhang, Ziyong Li, Wen Yang, Chunlin Leng 외

During the development of large language models (LLMs), pre-training data play a critical role in shaping LLMs' capabilities. In recent years several large-scale and high-quality pre-training datasets have been released …