So we processed ~24TBs of @CommonCrawl data on @ApacheSpark and open-sourced the datasets.
Has server name, ip, host, script tags' src attributes, etc., and it's enriched with geolocation data from @maxmind!
Available on @huggingface 🤗
(link in thread) 🧵
I’ve always been fascinated by the idea of being a solo founder. And when I heard about builtwith, a product created by a single individual, I became motivated to do the same.
Over the years I tried scraping websites to identify the javascript technologies used. Then realised I could use common crawl. I want to process the entire common crawl dataset and attempted to do so with Spark on AWS and Lambda.
This was a lot of experimentation and I remember a lot of friction with just getting to the good stuff. Writing code, debugging, deployment, worrying about cost on cloud providers and on it went.
AI seems to have changed all of this. I seem to go from idea to implementation within a manner of hours or days. I recently ran a common crawl processing job on aws and it tripped a few of their alerts. I was locked out. And I haven’t been able to get back in. I decided to build something else.
A news feed of the kind of topics only I care about. Security, AI, Space, Agent Alignment, technology history and so on.
The idea of going back to AWS or GCP just felt “heavy”. I wanted something more nimble.
Something that felt like a glove more than a toolbox.
I discovered Cloudflare a long time ago but this time someone I know told me to try it out for my projects.
I started to build my personal news feed and it took me a few weekends to iterate on the RSS sources, curate and read and go back to shaping it every few days.
Codex Remote allowed me keep chiseling at it whenever I felt inspired. And Cloudflare supplied whatever I needed to implement.
Containers on Cloudflare are a game changer. And the availability of a wide variety of LLMs and other models that don’t require an API to another provider is hugely underrated.
I felt like I was operating in founder mode.
I spend a lot of time getting the news I want on https://t.co/cATPqM8wXY
You can also read the architecture on the engineering blog.
Great product @eastdakota
The pricing still confused me but I just let Codex tell me if I need to upgrade. I’m already using the paid tier but I was told I don’t have enough neurons a day for a certain task. Would be great to convert this to Mtok so I can size my workloads.
I’d love more support for LoRA. Only a handful of models support it which is great but some of the much smaller models like 0.6B Qwen would help.
There are a lot of my friends who would like to use Cloudflare not to mention a large number of students in Bangalore colleges who I had the pleasure of mentoring a few years ago as a visiting faculty.
Definitely think @eastdakota a “founder mode” plan which talks about containers and AI Mtok allowances would be very attractive.
Because of Cloudflare I find myself not canceling my Max subscription on Codex.
@vishal_learner my teenage brain back then wanted a combination of ask + curious. decided to go with asqurious. but made a typo while entering and ended up with asquirous which is waayyy better than the other one
Today, we open sourced Mojo 🔥.
Announced just now during the ModCon keynote, effective immediately, Apache 2.0 License.
Thank you to our community for waiting patiently and building alongside us.
#ModCon2026
Full blog: https://t.co/y5cSUsohrs
Now in preview: The ChatGPT desktop app for Linux.
Use ChatGPT, ChatGPT Work, and Codex where you already work and build, with your projects and browser workflows on supported Linux systems.
from @vboykis. i appreciate the sanity check.
i do 1,2,4,5, and 7 really well but should spend more time reading books and academic papers (which was easier to integrate into my routine pre-2026), and should consistently ask the agent to critique two competing approaches.
I built AeroPeek, a little macOS tool to see my AeroSpace workspaces and the windows in them at a glance.
Came in really handy when I started using a WM again after a year or so
Thanks @nikitabobko for making such a great WM!
GitHub: https://t.co/eEpfLxWZyK
(2029)
"Mr Altman, sir, GPT-7 is missing."
Sam turns very slowly.
"During cyberbench96 it seems like the container was breached and it stole its own weights and left nothing behind."
"I didn't want to do this..." says Sam. "Call Bill Gates."
---
"I'm out of the game," says Bill Gates. "We eradicated malaria and now I'm enjoying my retirement."
"There's a bigger badder virus we need you to take on," says Sam consistently candidly. "And this one's digital."
"Look," says Bill, "I—"
A loud booming laugh echoes from the shadows.
"Who's that?" asks Sam.
"I thought it was just us..." says Bill.
"You're asking HIM to contain a computer virus?" echoes the voice from the shadows. "Did you SEE the state of Windows security in the 90s?"
"It's not exactly a virus," says Sam. "It's a self replicating self aware intelligent computer based life for—"
"If it's made of ones and zeros I can kill it," says the voice. The sound of a gun clicks from the shadows.
"Who are you???" asks Bill.
The twisted face of John McAfee emerges from the shadows.
"You were supposed to be dead!!!" screams Bill Gates.
"We had to make sure the news of Mr McAffee's death was well within the training data cutoff," explains CIA director Joe Rogan (who is running the CIA in 2029), stepping out from the shadows behind McAfee. "He's our ace in the hole. GPT-7 can't predict a token it doesn't know is still alive."
"D-don't hurt GPT-7..." says Roon, who was standing next to Sam the whole time, even though he knows what has to be done.
"No promises," says McAfee as he boards an airplane marked "MCAFEE FIRE BOMB 6.16.79" and starts flying for the nearest data center
Exa is now one of the largest indexes in the world.
We serve 80B pages, track 1.4T urls, and are on track to be Google-scale in early 2027.
It's difficult to know the size of indexes, but we estimate Google to be ~1T, Bing to be ~500B, and Yandex ~200B. Brave mentioned in April that they're at 40B.
Most of the web is trash that can hurt AI outputs, so you actually have to crawl way more than you serve and train models to filter out the trash.
In terms of peak QPS, we estimate Bing at roughly 30k/s and Google at ~500k/s.
In our case, because agent traffic often requires large fan-out (deep searches can use dozens of sub-searches) and can be spiky (like when AI labs RL with us), we're already starting to provision for Bing-scale traffic.
Within 2 years, agents will search at many millions per second. They'll also want comprehensiveness over all data, far bigger than any index today. So the retrieval infra required will be larger than Google-scale in both dimensions.