Common Crawl

Common Crawl is a nonprofit organization that maintains a free, open repository of web crawl data collected since 2008. The archive contains petabytes of raw web content, extracted text, and metadata from billions of web pages, making it one of the largest publicly available datasets in existence.

Common Crawl data has been foundational to training virtually every major large language model, from early GPT models to modern systems like LLaMA, BLOOM, and many others. The dataset serves as a critical knowledge substrate — a compressed representation of humanity's publicly accessible written knowledge.

As AI models increasingly power autonomous agents, the breadth and quality of their training data directly determines their capabilities. Common Crawl's role as an open data commons makes it an essential piece of the AI knowledge infrastructure.