ML//training//dataset//Common Crawl

Common Crawl is a public, petabyte-scale archive of recurring web crawls. It supplies raw pages, metadata, and text-oriented derivatives rather than a clean machine-learning dataset.

Common Crawl is a public, petabyte-scale archive of recurring web crawls. It supplies raw pages, metadata, and text-oriented derivatives rather than a clean machine-learning dataset.

It has been a major source for many language-model pre-training corpora, usually after aggressive language detection, quality filtering, deduplication, safety filtering, and document extraction.

The crawl reflects what is accessible and discoverable on the public web, not humanity in proportion. Ranking, geography, robots rules, repeated templates, SEO spam, and historical accidents all shape the sample.

Filtering is not neutral cleanup. Every heuristic changes which languages, writing styles, communities, and kinds of knowledge survive into training.

Common Crawl makes web-scale data available; the dataset people actually train on is the pipeline applied afterward.