Everything you need to know about llms.txt: what it is, how it works, what AI systems can use it for, and whether it’s actually worth adding to your site 👇
https://t.co/goTQxFxCto.
how to make your site readable by ai agents:
1. add an llms.txt so search tools stop guessing urls
2. check your server logs, not your analytics
3. serve a markdown version of every page
4. support content negotiation headers
5. rethink your JS-only widgets
The llms.txt file is a proposed standard meant to help large language models (LLMs) better understand and use content from websites.
The idea is pretty straightforward: Instead of letting AI crawlers wander around your site, you give them a curated list of your most important content. To tell AI what content on your site it should actually pay attention to.
Read the full breakdown: https://t.co/11CAd5iiJ9.
Today I accidentally discovered something interesting. 🤯
I was asking Cursor to work with the X API and noticed it first fetched:
/llms.txt
My reaction:
"What in the AI-generated nonsense is this?" 😂
Turns out it's basically a special file for AI systems.
Think of it like:
📖 robots.txt — for search engines
🤖 llms.txt — for AI models
It helps AI tools understand where the docs are, what pages matter, and how to navigate a website's documentation more efficiently.
The craziest part?
I've been building software for years and had never heard about it until today. 💀
The web is slowly getting routes designed specifically for AI, not humans.
What obscure developer thing did you discover embarrassingly late? 👇
#AI #LLMs #CursorAI #Programming #Developer #SoftwareEngineering #BuildInPublic #TechTwitter
Google has added an experimental #AgenticBrowsing audit to Lighthouse.
To try it:
Install Chrome Canary
Open any page
Open DevTools
Go to Lighthouse
Enable "Agentic Browsing"
Run the audit
The current report checks:
- Accessibility Tree structure
- Layout stability (CLS)
- llms.txt recommendations
- WebMCP-related checks (where applicable)
Worth spending 10 minutes with if you care about:
#TechnicalSEO #Accessibility #CoreWebVitals #AISEO
I'm collecting examples while testing it across SEO, publishing, documentation and AI sites.
Has anyone found a page that passes every available check yet?
llms.txt 其实对网站 GEO:
1、google 自己文档说没用,然后自己文档也有 llms.txt 【打架】
2、经过观察 ai bot 爬虫等数据测试,其实有用的。方便 ai bot 爬到你,引用你
3、「内容侧」本质还是什么时候有这个 llms.txt & 里面的内容如何从 web 内容转化出来