Progressive disclosure is not reliable because LLMs are inherently lazy.
"In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it."
Vercel ran evals on Next.js 16 APIs that aren't in model training data to test whether agents could learn framework-specific knowledge through Skills vs. persistent context.
Skills are the "correct" abstraction: package domain knowledge, let the agent invoke it when needed, minimal context. The agent decides when to retrieve.
They work well WHEN the user triggers them; otherwise, LLMs just ignore them.
Vercel's benchmarking is the first experiment of this kind I've seen, and it's actually interesting.
- Baseline (no docs): 53%
- Skill (default): 53%
- Skill with explicit instructions: 79%
- AGENTS[.]md with 8KB compressed docs index: 100%
The skill approach assumes agents reliably recognize when they need external knowledge and act on it. They don't.
"You MUST invoke the skill" made agents read docs first and miss project context. "Explore project first, then invoke" performed better. Same skill, different outcomes based on prompting.
The winning approach removed the decision entirely. An 8KB compressed index embedded in AGENTS[.]md, with one instruction: "Prefer retrieval-led reasoning over pre-training-led reasoning."
Two agent design learnings:
1. Passive context beats active retrieval for foundational knowledge. Don't make the agent decide to look things up, make the index always present.
2. Compress aggressively. Vercel went from 40KB to 8KB (80% reduction) with zero performance loss. The agent needs to know where to find docs, not have full content in context.
The gap between "agent can access X" and "agent will access X" is larger than we assume.
I keep seeing similar findings across agent architectures. Kimi Swarm's orchestrator is trained specifically to avoid sequential execution. Without training, orchestrators default to serial processing, planning a list of steps and executing them one by one. It's the EASY path.
The agent defaults to the lazy path: hallucinating from training data rather than retrieving docs. Passive context removes the choice entirely; the agent doesn't decide whether to look things up; the index is already there.
We keep finding that the "smarter", more autonomous design (let the agent decide when to X) underperforms the "dumber" design (always X, or structurally enforce X).
@xai@grok bug: https://t.co/EbejzSH5fH webapp freezes for a solid minute after pasting a long block of code, and it typing gets really sloppy after 10k-20k characters. same thing on opera & chrome.
yesterday, i stumbled onto the most underrated market research tool.
tiktok creator insights.
it's a goldmine of consumer behavior data, hiding in plain sight. and it's free to use.
here's why it's powerful:
1. shows you what people are desperately searching for
2. highlights topics with high demand but low supply
3. reveals trending questions in every industry
4. tracks search growth over 14-day periods
the "content gap" tab shows you problems people are actively trying to solve, but can't find good solutions for.
so that's cool for a couple reasons
1. help you create content that has low supply/high demand (better chances of going viral)
2. you can build startups to some of these trends
Example:
i searched "email management" and found:
• "how to clear 10k emails"
• "best way to organize work inbox"
• "email templates for busy people"
thousands searching.
hardly any solutions.
the beauty of this
• it's real-time market research
• it's actual user intent
• it's completely free
• and most founders aren't using it
a bunch of smart founders are mining tiktok insights right now
it isn't perfect, but you never know what you might find
your next startup idea might be hiding in those search trends.
So, ill share how to access it because it’s kinda hidden:
1. Go to TT search
2.Type in “creator search insight”
3. Tap view
im one of those people that think using data like this is your unfair advantage.
if tiktok is the new search engine, then tiktok creator insights is the new google trends.
might as well use it.
This @openai researcher got hacked. The classic "a coin just dropped" thing with comments turned off and a fake website. An analysis on how the attack works and its tech stack 🧵