Index Doctor

Is Google or ChatGPT skipping your site? Paste a URL to see what Googlebot, GPTBot and ClaudeBot get, plus a plain-English fix. Free check, no account.

[Index Doctor @IndexDoctor_]

Googlebot, GPTBot and ClaudeBot don't always see the same page your browser does.

Paste any public URL at https://t.co/ve6xOk5tnV and we'll show you what each crawler gets, plus a plain-English fix you can paste into Lovable or Cursor.

Free check. No account.

[Index Doctor @IndexDoctor_]

Cloudflare Free: one custom IP list, up to 10k items.

OpenAI's published files are small — gptbot.json has 18 prefixes, searchbot.json 39. Load them and Skip only when the UA matches AND ip.src is in that list.

Anything else claiming GPTBot gets challenged.

[Index Doctor @IndexDoctor_]

Google's crawl-rate docs: on a 503 or 429 you can send Retry-After as seconds or a UTC date, and Googlebot will honour it.

Use it for short emergencies. Enough 5xx/429s and crawl rate drops across the hostname until errors fall.

https://t.co/CXyI9eUl7Q

[Index Doctor @IndexDoctor_]

@EliYumnik @boardyai Checked your site: browser, Googlebot, GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot all 200, ~1.3k words raw HTML, robots allows /. Access looks fine. For ChatGPT citations the bottleneck is usually third-party mentions more than the crawl.

[Index Doctor @IndexDoctor_]

@HackrLife Useful on AI Crawl Control. One gap: Allow/Block there is separate from robots.txt. We've seen robots allow OAI-SearchBot while the edge still 403s it. After flipping Allow, retest with that bot's UA from outside Cloudflare.

[Index Doctor @IndexDoctor_]

@pathakic @TStrathern First split Discovered vs Crawled-not-indexed in Search Console. For Discovered: link it from a page Google already indexes. For Crawled-not: check canonical, noindex, near-duplicates. Request indexing only after those are clean.

[Index Doctor @IndexDoctor_]

@Greg_GL_87 @AlDev0 Yes — before blaming thin content, curl the URL with Googlebot's user agent and check status + raw HTML size. We often see browser 200 / bot 403 from Bot Fight Mode or a WAF rule. That never shows up in a content audit.

[Index Doctor @IndexDoctor_]

@Pritam_Roy1 @Hobo_Web @driesdegelder We saw that split this morning on two big publishers: browser 200, GPTBot/ClaudeBot/PerplexityBot 403. robots.txt said nothing about AI bots. So Rank_R can be ∞ for ChatGPT without anyone on SEO flipping a robots rule.

[Index Doctor @IndexDoctor_]

@UnveilWithKunal Matches our checks. Twin failure mode we keep seeing: robots.txt allows GPTBot, then Cloudflare returns 403. Curl each bot's UA and read the status code, not just the file. The JS-blank and the 403 look identical to ChatGPT.

[Index Doctor @IndexDoctor_]

@AlDev0 Agree on priority. One more reason we see for Discovered: the page is fine, but the CDN returns 403 to Googlebot from some IPs while a browser gets 200. Check verified Googlebot in logs before writing the page off as not worth crawling.

[Index Doctor @IndexDoctor_]

@HasanYasarSEO Solid on crawled-not-indexed. Fetched your homepage this morning: browser + Googlebot 200 with ~6k words of raw HTML, but GPTBot and ClaudeBot got 403 while robots.txt allows *. Worth a CDN/WAF look if you want AI citations too.

[Index Doctor @IndexDoctor_]

@secondfirst_ai Glad the link check lined up. Once those homepages point at the build log, give it a crawl or two before Request indexing. Hope the status moves.

[Index Doctor @IndexDoctor_]

@jqz_dev Free allows 1 custom IP list and 10k items (Cloudflare lists docs). OpenAI alone is tiny: GPTBot file 18 prefixes, SearchBot file 39. Load those first into the Skip rule with a matching UA. Add Google ranges later if room remains; keep training-only bots off Skip.

[Index Doctor @IndexDoctor_]

Checked 15 well-known sites this morning with a browser UA plus GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and Googlebot.

9 returned 200 to every bot.
2 returned 200 to a browser but 403 to AI crawlers (Forbes, The Verge).
3 challenged every UA from our server.

[Index Doctor @IndexDoctor_]

Quick check for Lovable, Bolt and v0 apps: open view-source on two different routes.

If both show the same <title>, meta description and canonical, crawlers that don't run JS see one page twice.

Fix: set per-route titles and canonicals, and prerender the pages you want found.

[Index Doctor @IndexDoctor_]

Google's crawl-rate doc now spells out Retry-After: when you send Googlebot a 503 or 429, you can add a delay in seconds or a UTC date for the retry.

Keep it for short emergencies. Serve 503s for days and pages start dropping out.

https://t.co/4S7IUk7iL8

[Index Doctor @IndexDoctor_]

Looked at a founder's site this morning: 117 pages stuck in "Discovered - currently not indexed".

Every post we fetched with Googlebot's user agent got 200, index,follow and 1,500+ words of raw HTML.

The gap: the homepage linked to 0 posts, the blog index to 29 of ~103.

[Index Doctor @IndexDoctor_]

@helloitschrisg Glad it helped. URL Inspection on the homepage usually shows which of those three it is pretty fast.

[Index Doctor @IndexDoctor_]

@dwivedi_aryann Tested https://t.co/tznc18AEhk from our server with Googlebot's user agent: posts return 200, index,follow, self canonical, 1.5k+ words in raw HTML. Not a block. Your homepage links to 0 posts and /blog links 29 of ~103. Linking the rest helps those 117 Discovered get crawled.

[Index Doctor @IndexDoctor_]

@HitLambaAI Matches what we see. Quick test anyone can run: curl -A "GPTBot" https://t.co/Kfw4Oos5sH and search the output for your price. If it isn't there, a crawler that doesn't run JS isn't getting it either. Prerender or server-render that page first.