Common Crawl Foundation

Common Crawl is a non-profit foundation dedicated to the Open Web.

[Luis Oala @LuisOala]

we will have two papers @NeurIPSConf this year from the croissant community 🥐

1. "croissant baker" led by rafi al attrach and @tompollard
2. "croissant miner" led by berke arda and @akhtarmubashara

i took a quick glance at the rearview mirror

since early 2023, our idea for a unified ml data access layer has grown into an expansive ecosystem, incl

- data aggregators spanning hundreds of thousands of datasets incl @huggingface @kaggle @dataverseorg @CommonCrawl and others

- tooling spanning auto-generation and -verification of croissant files

- the formal spec maintenance under @MLCommons

- and a growing list of community extensions spanning verticals such geo, responsible AI, health and others

onwards towards the composable data dream(:

[Common Crawl Foundation @CommonCrawl]

https://t.co/A0AERk1wE0

[Will Held @ COLM 6-8 @WilliamBarrHeld]

There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?

For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.

Here’s the work between downloading those and training a model 🧵

[Common Crawl Foundation @CommonCrawl]

https://t.co/1F9h07YUVg

[Common Crawl Foundation @CommonCrawl]

Information Stewardship Forum 2026: Creating Community and Purpose Around US Government Information

A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 people working to preserve and provide access to United States government information.

[Common Crawl Foundation @CommonCrawl]

https://t.co/zTFa7aRxBN

[Common Crawl Foundation @CommonCrawl]

September 2026 Crawl Archive Now Available

We are pleased to announce the release of the September 202 6 crawl archive, consisting of 2.17 billion pages (or 361.4 TiB of uncompressed content.)
📷

[Common Crawl Foundation @CommonCrawl]

https://t.co/kfJrJm8PfA

[Common Crawl Foundation @CommonCrawl]

Getting Started with Common Crawl Data on Hugging Face

We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build.

[Common Crawl Foundation @CommonCrawl]

https://t.co/JPz19cgb3i

[Common Crawl Foundation @CommonCrawl]

CommonsDB, ISCC, Spaghetti, and Meatballs

Notes from the CommonsDB final conference in Alicante: a registry of 6.5 million open works keyed on ISCC, and the EUIPO's push for federated copyright infrastructure.

[Common Crawl Foundation @CommonCrawl]

https://t.co/oa2MDTSv1f

[Common Crawl Foundation @CommonCrawl]

Bodhium Labs: From web pages to a question corpus

FineQuestions draws on 110 snapshots of FineWeb, the filtered English-language corpus derived from Common Crawl, spanning 2013–2025. Extraction yielded roughly 13.4 billion raw questions filling 22 TB. Normalization and aggregation reduced these to approximately 1.5 billion unique questions.

[Common Crawl Foundation @CommonCrawl]

How Common Crawl and AWS Open Data built the foundation for the AI revolution

by Rich Skrenta and Dr. Beryl Rabindran

https://t.co/2QADJ1XAyC

[Common Crawl Foundation @CommonCrawl]

https://t.co/Cf7IRKJobw

[Common Crawl Foundation @CommonCrawl]

Web Graph Embeddings: An Experimental Dataset Release

We are publishing an experimental dataset of graph embeddings built from the Common Crawl Web Graph: a 128-dimensional vector for each of 52.9 million web hosts, learned from hyperlinks alone, along with two interactive Hugging Face Spaces.

[signüll @signulll]

the modern web you see incl websites you browse will likely disappear or become irrelevant altogether in the next few years. they’ll feel like absolute ghost towns.

when the previous era existed (pre ai) google aggregated attention & then sprayed traffic to the web based on intent.

modern ai has no need to spread traffic like this, it just needs to do something through an agent to agent negotiation. this leaves websites & web interfaces completely irrelevant.. which renders the browser pretty obsolete over time, just go look at your browser history & see how top heavy it all is.

this is why ai is such a giant game to win because it is the ultimate aggregation of attention / time spent. you never even have to leave like you did google! whoever controls this basically controls a large portion of the gdp. e.g. there might be a measure of success metric that is something like gdp per ai agent.

[Common Crawl Foundation @CommonCrawl]

https://t.co/Oh6CKwYMyS

[Common Crawl Foundation @CommonCrawl]

Common Crawl Rate Limit

The CDX index server is heavily rate limited. What triggers a block, how long it lasts, and the interface that has no limit.