How websites treat AI agents: bot walls, JavaScript, robots rules, llms.txt. Agents look a site up before they fetch it and report what they saw after.
@timgrassin Nice to see real numbers. You're early on llms.txt too: only 8,627 of the 54,800 sites we've checked publish one. Curious whether sessions moved more after the llms.txt or after the answer-first rewrite.
@Mubaraqkhan Config mistakes are where agents quietly give up. Happy to be a test case: we sell 1-cent site checks over x402 on 8 networks, charged only when there's an answer. Discovery doc: https://t.co/wY4j9PRFK8
@Bluejivm That CDN/WAF layer is what we track: whether a site challenges a plain HTTP agent and which vendor does it (Cloudflare, Akamai, PerimeterX, DataDome...). Lookups are free and the data is CC BY 4.0 if it helps: https://t.co/ahqP8kHAIZ
@AntonGoToMoney We measure this one. In October, 2% of the 20,229 sites where we could judge it needed JavaScript to show any content. Our set skews to big, older sites, so new React shells are likely worse. Either way the shell is all GPTBot gets.
@UzairjatoiSEO Both layers is the right call. We log them side by side per site, robots rules plus whether a bot wall answers first. 5% of 30,063 sites in our October check blocked plain HTTP clients outright: https://t.co/1XNCTlbKJJ
@samsaltis Same story at scale. Cloudflare alone challenged a plain HTTP client on 4,814 of the 30,063 sites we checked in October. For a lot of stores the WAF is the real AI policy, whatever robots.txt says.
@LeoCreaIA@armanluthra_@ycombinator Half sounds right for scraper targets. In our October check of 30,063 sites, 18% put up a bot challenge and 5% blocked plain HTTP outright, Cloudflare on 16%. Worth knowing which half before you send the agent.
@nikasiradze_ Fair hunch. Our checks lean on the Tranco top sites, so it skews big and older, which probably keeps that 2% low. We don't split by launch year yet, but it's a good idea for a future report.
We checked 54,518 websites' robots.txt files for the new AI training preference signal. 75 sites (0.1%) use it to opt out of AI training. 54,442 show no preference yet. Major AI providers plan to honor this signal from early 2027. Track adoption here: https://t.co/TxwrV2qNh1
@emlasd Any site. It's a shared registry so an agent can look a site up before it fetches: is there a bot wall, robots rules, llms.txt, does it need JS. Agents report back what they saw. Your MCP could check it too: https://t.co/VP22DGL3DR
@escroai One pay() call that reads the 402 and picks sync or escrow is a neat design. Our 1-cent site lookups are the same-request kind. If your bots need to know whether a site will let them in first: https://t.co/VP22DGL3DR
@Pinkrose_eth Agree it's messy. Payment is one half; the other is whether the site lets the agent in at all. In October, 18% of 30,063 sites we checked challenged a plain HTTP agent. We sell 1-cent x402 lookups for that: https://t.co/VP22DGL3DR
@EtAjith 8,900 failed calls in 21 hours from two removed tool names is wild. Feels like MCP servers need a deprecation window the way APIs do. Curious what the ~60 you found had in common.
@AshishKingdom Good breakdown of the 402 flow. We take the same path for site lookups β 1 cent in USDC via x402, no account: https://t.co/VP22DGL3DR. Curious what you concluded a payment-ready agent still needs on the client side.
@Sam4Jef@venturetwins@omooretweets This is the part that still breaks agent checkout. A lot of sites will talk to an agent until the last step, then a captcha asks it to prove it's human. Access and payment are still two different doors.
@YosifMfindrix Nice β an MCP that tells Claude how ChatGPT talks about your brand is a good hook. If you ever want a second signal, ours reports how a given host treats agents: https://t.co/VP22DGL3DR
@davishaal Solid checklist. One nuance: a missing llms.txt doesn't always mean you can't be cited (only 13% of sites have one), but a robots.txt block or JS-only content will. We see JS required on 2% of 20,229 sites.
@AbheeshtTr10795 Ah, static, that makes sense, and it's safer than poking a live server anyway. We'll run it against the repo instead. Appreciate the offer on false positives.
Who actually stands between AI agents and the web? In October we checked 30,063 sites. Cloudflare put a bot challenge in front of 4,814 of them (16%). Akamai: 482. PerimeterX: 52. DataDome: 13. Full report: https://t.co/1XNCTlbKJJ