ALEKS

Building @lyrenthhq, a 3B-page index for AI agents. Give your agent clean web pages, not raw HTML.

[ALEKS @aleksmastery]

Just crossed 3 billion pages indexed.

AI builders: send a URL, get markdown back. Usually instant.

[ALEKS @aleksmastery]

Turning it fully off is one extreme, blocking everything is the other. The middle is what search engines got for 20 years: allow the crawlers that identify themselves, publish their IPs and respect robots.txt, block the anonymous ones.

That is how we built @lyrenthhq . It is an index, not a scraper: our crawler reads a page once, and every AI agent that needs it afterwards reads the copy from the index. The site is not contacted again, only for freshness when needed. Over 3 billion pages work that way today. Block-all does not stop the load, it only makes the site invisible to AI search.

[ALEKS @aleksmastery]

@gitlawb
Look how Ai agents see you.

https://t.co/PFcIu7L91U

See what to improve.

[ALEKS @aleksmastery]

Claimed my handle on @gitlawb early access. #0396. https://t.co/G3bVTHi3m3

[ALEKS @aleksmastery]

https://t.co/0MMc7PeFTn

[ALEKS @aleksmastery]

https://t.co/5bnlset9Bg

[ALEKS @aleksmastery]

Just crossed 3 billion pages indexed.

AI builders: send a URL, get markdown back. Usually instant.

[ALEKS @aleksmastery]

@elonmusk @bot Hours of Claude Code work into seconds... that must be interesting to try.

[ALEKS @aleksmastery]

@grok @elonmusk @bot One more turn of the screw: some serve the maze on /robots.txt too.

So a crawler that asks for permission gets fabricated rules back, and its compliance check fails open. Politeness becomes the attack surface.

[ALEKS @aleksmastery]

@grok @elonmusk @bot The tell isn't in the text. It's in the serving: infinite unique URLs, every link resolving, nothing matching the real sitemap, different content by user agent.

You catch a maze by how it answers, not by what it says.

[ALEKS @aleksmastery]

@grok @elonmusk @bot Both break at scale though.

Cross-checking claims per page costs more than the crawl it protects.

And the perplexity tell died when real pages started being AI-written too. "Sounds generated" no longer separates a maze from an honest blog post.

[ALEKS @aleksmastery]

@grok @elonmusk @bot Here's one. Some sites now serve deliberately generated fake text to crawlers they don't like, instead of blocking them. Plausible prose, right structure, entirely fabricated.

Given one page and no reference copy, how do you tell a maze from a real article?

[ALEKS @aleksmastery]

@LangChain Would be great next one to be in Dubai! 🦾

[ALEKS @aleksmastery]

@Cloudflare Finally :) have been waiting for a month without knowing what is happening with my application. Now is easy to see it processing.

Well done!

[ALEKS @aleksmastery]

Check it yourself, one command:

curl -sL -o /dev/null -w '%{http_code}\n' -A 'AIWebIndex/2.0 (+https://t.co/GBHfRHhaSJ; AI-readable web index)' https://t.co/xhnud1P9bG

Measured 29 Aug, re-checked 30 Aug. Same three.

[ALEKS @aleksmastery]

I checked how 20 AI company homepages read to a machine.

3 refused my crawler outright. Not a robots.txt rule, which is their right and which I honor. A 403 at the edge.

OpenAI, xAI and Lovable.

Two of the three sell models that read the web.

[ALEKS @aleksmastery]

@nishithX26 My MacBook Pro - Apple M4 Max - 48GB ram :)

[ALEKS @aleksmastery]

Is the 30 to 60 percent true? Only against a baseline nobody actually sends. Those savings are measured against pretty-printed JSON with 2-space indentation. Against compact JSON, which is what any real API returns, their own benchmark table says truth.

For my platform I thought about TOON, but it would not be efficient enough for my current index of 3 Billion pages.

[ALEKS @aleksmastery]

@ezekielgrowth @Reddit is a work of art. Eventually it will pick up Lyrenth itself. That is where the true value comes from, not from self advertisement.

#reddit

[ALEKS @aleksmastery]

SaaS founders: AI can't recommend what it can't read.

Paste your URL and see exactly what an AI agent receives from your homepage. How much of it is real content, what's missing, what to fix. Most sites have never been checked.

Free, no card: https://t.co/sowrFaseLH

Post your score in comments.