Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality set. Importantly, we find that training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large. This finding alone is particularly striking, given the substantial effort and cost recently devoted to augmenting high-quality data. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well as the low-quality one. Finally, we introduce an optimization-driven notion of data quality and demonstrate that it can be reliably estimated using small-scale proxy experiments. Altogether, our results both elucidate the mechanisms behind CQF and deepen our understanding of data selection methods widely used in practice.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SSC-UNet: UNet with Self-Supervised Contrastive Learning for Phonocardiography Noise Reduction
Congenital Heart Disease (CHD) remains a significant global health concern affecting approximately 1\% of births worldwide. Phonocardiography has emerged as a supplementary tool to diagnose CHD cost-effectively. However,…
Contrastive LearningUnderstanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought
During both pretraining and fine-tuning, Large Language Models (\textbf{LLMs}) are trained on trillions of tokens of text of widely varying quality. Both phases of training typically involve heuristically filtering out `…
Real-time Local Noise Filter in 3D Visualization of CT Data
Removing noise in computer tomography (CT) data for real-time 3D visualization is vital to improving the quality of the final display. However, the CT noise cannot be removed by straight averaging because the noise has a…
A Survey on the Visual Perceptions of Gaussian Noise Filtering on Photography
Statisticians, as well as machine learning and computer vision experts, have been studying image reconstitution through denoising different domains of photography, such as textual documentation, tomographic, astronomical…
DenoisingSurveyDPSR: Differentially Private Sparse Reconstruction via Multi-Stage Denoising for Recommender Systems
Differential privacy (DP) has emerged as the gold standard for protecting user data in recommender systems, but existing privacy-preserving mechanisms face a fundamental challenge: the privacy-utility tradeoff inevitably…
Collaborative Filtering