ML//training//dataset//Common Crawl

Common Crawl is a public, petabyte-scale archive of recurring web crawls. It supplies raw pages, metadata, and text-oriented derivatives rather than a clean machine-learning dataset.

Common Crawl is a public, petabyte-scale archive of recurring web crawls. It supplies raw pages, metadata, and text-oriented derivatives rather than a clean machine-learning dataset.

It is run by a non-profit organization that has crawled the web regularly since 2008 and gives the results away; each crawl holds a few billion pages, collected by crawlers that follow links and respect robots rules. Because the crawls are kept, the archive is also a record of what the web looked like at each date, useful for research on how it has changed, though no crawl is a complete copy of the internet.

It has been a major source for many language-model pre-training corpora, usually after aggressive language detection, quality filtering, deduplication, safety filtering, and document extraction.

The crawl reflects what is accessible and discoverable on the public web, not humanity in proportion. Ranking, geography, robots rules, repeated templates, SEO spam, and historical accidents all shape the sample.

Filtering is not neutral cleanup. Every heuristic changes which languages, writing styles, communities, and kinds of knowledge survive into training.

Common Crawl makes web-scale data available; the dataset people actually train on is the pipeline applied afterward.