Common Crawl Foundation

Common Crawl is a non-profit foundation dedicated to the Open Web.

[Common Crawl Foundation @CommonCrawl]

https://t.co/soTCxiDa2F

[Common Crawl Foundation @CommonCrawl]

October 2026 Crawl Archive Now Available

We are pleased to announce the release of the October 2026 crawl archive, consisting of 2.16 billion pages (or 371.19 TiB of uncompressed content)

[Common Crawl Foundation @CommonCrawl]

https://t.co/73XeMi7SOl

[Common Crawl Foundation @CommonCrawl]

https://t.co/CP9eOho78D

We present the results of the WMT 2026 shared task of the Open Language Data Initiative (OLDI). Participants were invited to contribute to the massively multilingual open datasets supported by OLDI, FLORES+ and OLDI Seed, or to create new resources in line with OLDI’s ethos. We accepted ten submissions: five extending FLORES+, and five extending other open and massively-parallel datasets (BOUQuET, Seed, SMOL, and WMT24+). These contributions advance the coverage and quality of multilingual datasets, especially for underserved language varieties. All contributions are released under permissive open-source licenses.

[Common Crawl Foundation @CommonCrawl]

Findings of the WMT 2026 Shared Task of the Open Language Data Initiative

Laurie Burchell Common Crawl Foundation

[Brian Wesley @BrianWesley0711]

@CommonCrawl My bet: low-resource languages would lose at the filter.

Quality classifiers trained mostly on English throw out good Icelandic pages as noise. Smarter crawl budgets won't help if the filter can't read the language.

[Common Crawl Foundation @CommonCrawl]

https://t.co/gM1lHljcHa

[Common Crawl Foundation @CommonCrawl]

The Common Crawl Foundation at the TPDL Conference

The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.

[Common Crawl Foundation @CommonCrawl]

https://t.co/m5DSDq5Koz

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages.

We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data.

[Common Crawl Foundation @CommonCrawl]

Common Crawl - the ultimate time capsule for knowledge | Amazon Web Services

https://t.co/gM7XAFDMhe

[Luis Oala @LuisOala]

we will have two papers @NeurIPSConf this year from the croissant community 🥐

1. "croissant baker" led by rafi al attrach and @tompollard
2. "croissant miner" led by berke arda and @akhtarmubashara

i took a quick glance at the rearview mirror

since early 2023, our idea for a unified ml data access layer has grown into an expansive ecosystem, incl

- data aggregators spanning hundreds of thousands of datasets incl @huggingface @kaggle @dataverseorg @CommonCrawl and others

- tooling spanning auto-generation and -verification of croissant files

- the formal spec maintenance under @MLCommons

- and a growing list of community extensions spanning verticals such geo, responsible AI, health and others

onwards towards the composable data dream(:

[Common Crawl Foundation @CommonCrawl]

https://t.co/A0AERk1wE0

[Will Held @WilliamBarrHeld]

There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?

For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.

Here’s the work between downloading those and training a model 🧵

[Common Crawl Foundation @CommonCrawl]

https://t.co/1F9h07YUVg

[Common Crawl Foundation @CommonCrawl]

Information Stewardship Forum 2026: Creating Community and Purpose Around US Government Information

A report from the inaugural Information Stewardship Forum, a three-day convening hosted by the Internet Archive in San Francisco on March 18–20, 2026, that brought together 120 people working to preserve and provide access to United States government information.

[Common Crawl Foundation @CommonCrawl]

https://t.co/zTFa7aRxBN

[Common Crawl Foundation @CommonCrawl]

September 2026 Crawl Archive Now Available

We are pleased to announce the release of the September 202 6 crawl archive, consisting of 2.17 billion pages (or 361.4 TiB of uncompressed content.)
📷

[Common Crawl Foundation @CommonCrawl]

https://t.co/kfJrJm8PfA

[Common Crawl Foundation @CommonCrawl]

Getting Started with Common Crawl Data on Hugging Face

We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build.