Agents Week at Cloudflare starts today. We're shipping the infrastructure for what comes after the browser -- the tools, protocols, and payment rails that let AI agents interact with the web as first-class citizens. Not just crawling it. Transacting on it. https://t.co/871HxBQbhE
we're not joking when we say our /crawl does not bypass Cloudflare's bot detection or Captchas
/crawl identifies as a bot + respects robots.txt, which is that standard used by websites since (checks notes) 1994 to indicate to crawlers/bots which parts of a website they are allowed to visit.
see the blocking in action below
People are completely missing the point of this feature. Most accidentally and probably some on purpose.
They stated very clearly that this obeys all blocking and content rules for your site. It’s literally the opposite of bypassing those controls.
The purpose of the system is to get rid of all the halfassed AI crawlers all over the Internet that are doing a crappy job of pulling your content.
They are loud, rude, and wasteful. And they’re filling the internet and all our logs with massive amounts background noise.
Cloudflare knows they are going to crawl no matter what.
This feature is simply giving them a legit way to do it efficiently so that they don’t clog up the entire Internet doing it in a way that it’s 1000x less efficient.
And it’s still uses all your rules for what can and cannot be crawled. And all of your other Cloudflare controls around crawling are still enforced.
If this is adopted to any significant degree, Cloudflare will be absolute heroes for reducing terabytes of crawler-slop background noise.
Saw a lot of misinformation around the new /crawl endpoint @CloudflareDev announced yesterday.
Go try scraping https://t.co/dvC9KZiY6k with the /crawl endpoint. We said we are going to protect your site, and we stand by it 🧡
The web was built for humans reading HTML. AI agents are drowning in error pages meant for browsers. @Cloudflare just shipped RFC 9457-compliant structured error responses: Markdown and JSON instead of heavyweight HTML. 98% fewer tokens wasted.
https://t.co/J4glEMhvs3
At @Cloudflare we will continue to build for AI Developers and Content Creators. We need to get the future economy of the internet right, and that means building ways to help AI *responsibly* ingest content in accordance with site owner's wishes!
We want to clarify something: one of the motivations for building this feature is that building a crawler that respects crawling best practices is hard. Too many crawlers use this as an excuse not to abide by site owners' preferences, and respect directives like robots.txt.
/crawl does this out of the box.
Additionally, the /crawl endpoint cannot bypass Cloudflare's bot detection or Captchas, and self-identifies as a bot. It respects all the same protections available to operators today: AI Crawl Control, robots.txt, Content Signals and Pay-Per-Crawl.
So, site owners can and should still choose how their content is consumed, and our /crawl endpoint will respect it.
If AI agents consume 50% of web traffic but pay 0% of the cost to create the content they consume, who keeps making content? This is the question the entire internet needs to answer in 2026.
The 2026 #F1 regulations are the biggest shake-up ever.
New active aero, new power units, completely rewritten rules.
If you're not watching the season opener this weekend, you're missing the best engineering competition on the planet.
https://t.co/RbwimBJLHd
Cloudflare acquires Human Native, an AI data marketplace specializing in transforming content into searchable and useful data, to accelerate work building new economic models for the Internet. https://t.co/yvwoA6lwbK
Human Native 🤝Cloudflare
We are thrilled to announce that @humannativeai has been acquired by @Cloudflare !
Check out the full story: https://t.co/8UtVv4VLED
@OfficialFPL Trying to setup my team and it's asking me to update my password. When I type in a new one it gives an error: "New password cannot be the same as your current password" - this is a new randomly generated password it's definitely not one I've used before. Help please!
Google just dropped a 100% free learning path on Generative AI with 9 Courses 👇
Intro to Gen AI
Intro to LLMs
Intro to Responsible AI
Intro to Image Generation
Encoder-Decoder
Attention Mechanism
Transformers and BERT
Image Captioning
Gen AI Studio
https://t.co/cyojw6P4Tm
28 short pieces of life advice:
1. Block off 90 minutes in your calendar every morning to work on the most important thing. Wake up early if you need to. Don’t compromise.
That time when we showed the Queen how to get driving directions to Windsor Castle on an early version of Google Maps for phones. The Queen was super curious and engaging. My first & last royal focus group! Circa 2008.