What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of …
Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails
Modern language models rely on pretraining filters to remove undesirable content from training corpora and inference-time guardrails to suppress undesirable outputs during deployment. In this paper, we examine how these …
Bias Analysis and Mitigation through Protected Attribute Detection and Regard Classification
Large language models (LLMs) acquire general linguistic knowledge from massive-scale pretraining. However, pretraining data mainly comprised of web-crawled texts contain undesirable social biases which can be perpetuated…
AttributeBuilding a User-Generated Content North-African Arabizi Treebank: Tackling Hell
We introduce the first treebank for a romanized user-generated content variety of Algerian, a North-African Arabic dialect known for its frequent usage of code-switching. Made of 1500 sentences, fully annotated in morpho…
Dependency ParsingPOSPOS TaggingSentence+1Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
When trained on large, unfiltered crawls from the internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: they often generate racist, sexist, violent or otherwise to…
Language ModelingLanguage Modelling