Most people made one decision about AI crawlers: block them all, or allow them all.
There are three kinds. They are not interchangeable.
https://t.co/8wtiVxOtPw BOTS — GPTBot, ClaudeBot, Google-Extended, CCBot
Your content feeds the next model. You get nothing back. Blocking these is a legitimate policy call and costs you almost no visibility.
https://t.co/w1zzyOVKp9 INDEX BOTS — OAI-SearchBot, Claude-SearchBot, PerplexityBot
These build the index the AI pulls from when someone asks a question. OpenAI’s own docs say sites blocking OAI-SearchBot won’t appear in ChatGPT search answers.
3.USER-TRIGGERED FETCHERS — ChatGPT-User, Claude-User
Someone asked the assistant to go look at your page, right now. Worth knowing: OpenAI and Perplexity both warn that robots.txt may not govern these the way it governs their automated crawlers.
Here’s the part that gets people:
Blocking the training bot does not block the search bot. Different user agent, different line in your file, completely different consequence.
The data shows most sites haven’t caught up. Around 69% block ClaudeBot and 62% block GPTBot, but only ~49% block OAI-SearchBot. Hostinger looked at 66.7 billion bot requests and found search-crawler access climbing from 4.7% to over 55% of sites while training-crawler access fell from 84% to 12%.
Translation: the people paying attention are splitting the difference. Everyone else made one blunt call two years ago and never revisited it.
So you get sites that blocked everything in 2024 and now wonder why they never get cited. And sites that allowed everything, got trained on for free, and got zero citations for it.
Opposite mistakes. Same single line of text.
Go open your robots.txt right now. If it’s one blanket Disallow, you made a choice you probably didn’t mean to make.
If you don’t want to parse your own file, I built a free checker that reads your robots.txt and tells you which of the three tiers you’re actually blocking → https://t.co/SGR9Yb5rtd
Most people made one decision about AI crawlers: block them all, or allow them all.
There are three kinds. They are not interchangeable.
https://t.co/8wtiVxOtPw BOTS — GPTBot, ClaudeBot, Google-Extended, CCBot
Your content feeds the next model. You get nothing back. Blocking these is a legitimate policy call and costs you almost no visibility.
https://t.co/w1zzyOVKp9 INDEX BOTS — OAI-SearchBot, Claude-SearchBot, PerplexityBot
These build the index the AI pulls from when someone asks a question. OpenAI’s own docs say sites blocking OAI-SearchBot won’t appear in ChatGPT search answers.
3.USER-TRIGGERED FETCHERS — ChatGPT-User, Claude-User
Someone asked the assistant to go look at your page, right now. Worth knowing: OpenAI and Perplexity both warn that robots.txt may not govern these the way it governs their automated crawlers.
Here’s the part that gets people:
Blocking the training bot does not block the search bot. Different user agent, different line in your file, completely different consequence.
The data shows most sites haven’t caught up. Around 69% block ClaudeBot and 62% block GPTBot, but only ~49% block OAI-SearchBot. Hostinger looked at 66.7 billion bot requests and found search-crawler access climbing from 4.7% to over 55% of sites while training-crawler access fell from 84% to 12%.
Translation: the people paying attention are splitting the difference. Everyone else made one blunt call two years ago and never revisited it.
So you get sites that blocked everything in 2024 and now wonder why they never get cited. And sites that allowed everything, got trained on for free, and got zero citations for it.
Opposite mistakes. Same single line of text.
Go open your robots.txt right now. If it’s one blanket Disallow, you made a choice you probably didn’t mean to make.
This isn't theory for me — it's my whole business. Solo founder, 20+ apps shipped, almost all of them running on open models via OpenRouter. The economics literally don't work at frontier API pricing for a one-man shop.
Open models are why indie builders can even exist right now. The Mac Mini era is next.
The greatest ever has joined X, and his first post is a banger
A letter on why open models are the future
He’s 100% right. And you should be paying attention
Open models will allow every company and person have their own custom, Fable level intelligence. Built completely for their needs
Fine tuned to do exactly what they need it to do
All data, prompts, tokens, completely private
Running 100% on their desks on something as small as a DGX Spark or Mac Mini
This future is guaranteed to come. Every path is leading this way. No government or corporation will be able to stop it
Open models are getting exponentially better and more efficient
They have caught up to frontier, and soon their tokens will be so efficient to create you’ll be able to run them on crap hardware
The time to start getting ready, educating yourself, and even setting up these models is now
It doesn’t matter what hardware you have. You can do this.
All you need is a Hermes or OpenClaw to be your local AI administrator
Ask it to examine your hardware, then look at Hugging Face and see which models you can run. Then ask it to list them with top use cases you can run on each.
Ask it to load the model and get started
Will they be the best models? Depending on your hardware probably not. But they’ll only get smarter and faster. This is the worst they’ll ever be.
The future is coming, and now one of the greatest entrepreneurs of all time came to this platform to reinforce it
Start preparing.
Right now, someone is asking ChatGPT to recommend a business like yours.
AI search doesn’t take days off. It’s answering right now — with you, or with your competitor.
Find out which in 60 seconds → https://t.co/SGR9Yb5rtd (free audit)
Built this solo from the Sierra foothills. If you run a local business or an agency, the audit takes 60 seconds and it’s free. Would love feedback from other builders.
My vinyl booth is in a town of 1,500 people.
When a tourist asks ChatGPT “where to buy records near Yosemite” — it either gets recommended or it’s invisible.
That’s the new Google.
I built a free tool that checks if AI can see YOUR business in 60 seconds → https://t.co/9chJxioWta
I turned the playbook into a tool: https://t.co/SGR9Yb5rtd
Paste your URL, get your llms.txt, robots.txt, schema + a visibility score.
But free first step: ask ChatGPT or Perplexity who they recommend in your niche.
If it's not you — that's fixable.
My agency’s website was 2 months old when Perplexity started recommending it by name.
No ads. No backlinks. No reviews.
Meanwhile Google Maps had me ranked below a San Diego agency with 40 reviews.
Same site. Two search worlds.
AI search is the flattest playing field in 15 years. Receipt below 👇
What the AI engines rewarded:
→ robots.txt that welcomes AI crawlers
→ llms.txt — a plain-text briefing on what the business does
→ JSON-LD schema so machines can parse the site
→ plain-English city + service pages
Google SEO takes years. This took a weekend.
If you’re trying to quit drinking, I built an app just for you.
It’s called Dry Day. First month’s on me.
Code DRYFAM at https://t.co/mPz63x3cDm
23 years sober. Don’t drink today. 🙏
That voice isn’t weakness — it’s the way out. Mine showed up 23 years ago and I finally listened. If today’s your day one, drop it below 👇 https://t.co/4YyNZhCmbw
That voice telling you “I can’t keep doing this”?
Listen to it.
That’s not weakness — that’s the way out.
23 years sober. Don’t drink today. 🙏 https://t.co/4YyNZhCmbw
This weekend I’m filming short-form for Dry Day AI — a sobriety app I built after losing my brother to alcohol. Turns out my real face beats every AI persona I tested by a mile 😂 Also shipping AlphaWhale AI, a distribution autopilot for app builders. Solo founder, ~20 products. What are you on?
23 years sober and here’s the math nobody mentions: the money you don’t hand over for booze starts buying you a better life. Tonight it’s steak. 🥩 https://t.co/4YyNZhCmbw
Walking into the store and not spending a dime of it on booze.
That’s $100+ going toward steak instead. 🥩
23 years sober. Don’t drink today guys. 🙏 https://t.co/4YyNZhCmbw