xulfi

Building tools for the agentic future 🤖 Recovering ice cream addict 🍦

[xulfi @xulfi0]

I checked the robots.txt of 945 well known websites for 14 AI crawlers. Of the 746 I could read, 193 (26%) ask at least one AI crawler to stay out. News leads: 80% of US news sites and 76% of UK ones. 🧵 https://t.co/NtWXIiHwgS

[xulfi @xulfi0]

@gauraavvii Autonomous agents

[xulfi @xulfi0]

Week 1 of building tools for the agentic future, real numbers: about 60 page views, 1 scan, 0 reports emailed, 0 paying customers. Also built;: 117 automated tests, a 945-site AI crawler study, 6 free tools, a supplier quote tool in pilot. Slow start, sharing it anyway.

[xulfi @xulfi0]

@itsankitakar There’s still great businesses solving real use cases on the smaller scale

[xulfi @xulfi0]

@pixel2ai Tips?

[xulfi @xulfi0]

@0xPrajwal_ What if you’re all three

[xulfi @xulfi0]

@Saanvi_dhillon Approving login to laptop

[xulfi @xulfi0]

@APompliano Carrie

[xulfi @xulfi0]

@devbyak If it shows in JS analytics, the visitor runs JavaScript, so it’s more likely headless browsers or agent tools than the big training crawlers. Segment by browser, country and very short ‘direct’ sessions, then compare with server logs by user agent. Curious what the split is.

[xulfi @xulfi0]

@autobloggingai Matches the llms.txt picture: cheap to add, little sign anyone fetches it. Useful data. Did you see any difference between bulk training crawlers and user-triggered fetches (the ChatGPT-User kind) on the .md files? That seems like the number that matters.

[xulfi @xulfi0]

@_alphashark_ @ZypherHQ Agree distribution is the hard part. One new wrinkle: for some buyers the first reader of your marketing is an agent. The same page that works for a person (plain facts, prices in the text, not blocked) is also what it can read. Human first, machine-readable second.

[xulfi @xulfi0]

@arberx_ That dance shows in the data. 78 big sites turned away my GPTBot/ClaudeBot test request while their robots.txt allowed all 14 crawlers. Bot protection and robots.txt often disagree, so someone has to own both. (My test came from Cloudflare’s network, so real crawlers may differ)

[xulfi @xulfi0]

@MrPythonic “By default” is doing a lot of work. I checked robots.txt on 945 big sites: of 746 readable, 26% block at least one AI crawler by name. Separately, 128 of 924 turned away a test request sent as GPTBot or ClaudeBot, whatever robots.txt said. Two different doors.

[xulfi @xulfi0]

@finalcall000 Allowing all of them on our own site. We build AI-readiness tools, so blocking would be odd, and nothing there is worth hiding from a model. Publishers are where I see most blocking (59 of 74 US news sites I checked), but I only know what the file says, not why. What’s yours?

[xulfi @xulfi0]

@atp_studio_coll No. The 193 are sites whose robots.txt I could read that block at least one AI crawler. The 199 that refused my request (127, mostly 403s), errored (51) or didn’t answer (21) are left out, not counted either way. The 128 in post 6 is a separate test: the homepage, not robots.txt

[xulfi @xulfi0]

I checked the robots.txt of 945 well known websites for 14 AI crawlers. Of the 746 I could read, 193 (26%) ask at least one AI crawler to stay out. News leads: 80% of US news sites and 76% of UK ones. 🧵 https://t.co/NtWXIiHwgS

[xulfi @xulfi0]

Look up any of the 945 (each has a page and a badge), download the data, or check your own site live: https://t.co/NtWXIiHwgS AI wrote most of the code; I checked the odd results.

[xulfi @xulfi0]

199 sites wouldn’t give my checker their robots.txt (bot protection, errors), so they’re left out. Method: homepage rules, read as RFC 9309 says, 21 to 22 Oct. It shows what sites ask for, not what AI companies do.