Software Engineer @Google
Author @freeCodeCamp
๐บ๐ธ.then(๐ช๐จ).then(๐จ๐ฑ).then(๐ง๐ท).then(๐ฉ๐ช).then(๐ฌ๐ง).finally(๐)
Thoughts are my own
I'm about to try something new. An experiment.
Can I create a new medical device from scratch? Hardware and all?
Can I do it solo on nights and weekends? Can I get it FDA cleared? How much can agents help? How long will it take? What will it cost?
Follow me on this journey.
When a new model comes out, are your instructions and skills still helping?
I wrote a walkthrough showing how to use DeltaSelect and Agent Layer to compare your setup to a bare model, based on the case study in my arXiv paper (https://t.co/IiBfM86tkh) https://t.co/M3Cgu8dH2F
My first @arxiv publication is live! DeltaSelect: Affordable A/B Testing for Coding Agents (https://t.co/IiBfM86tkh)
Use it with Agent Layer to compare harnesses, skills, and more. See if your setup is better than a bare model. More coming soon.
https://t.co/3JJSp2Vx0a
Simplify your AGENTS.md and skills. A two-line edit cut costs by 36.5% (p=.034), with a slightly better score on a four-task subset of DeepSWE by @datacurve. We have been told by @karpathy, @rubenhassid, @AnthropicAI, @OpenAI, and others to simplify. Listen to them.
I've been benchmarking my skills to try and do this in a principled way.
During benchmarking, I found that for hard tasks, the agent would review the plan and the code multiple times. It tended to continue beyond the point of diminishing returns, thinking that its findings were substantive when they were not.
So I made two minor changes to my skills that I have already significantly downsized.
Before:
> Update the plan to address any findings you agree with.
After:
> Update the plan to address any findings you agree with. Do not repeat plan review.
Before:
> Resolve any gaps, repeat any invalidated review, and run required checks before reporting completion.
After:
> Resolve any gaps and run required checks before reporting completion.
I tested both versions with GPT-5.6 Luna at low reasoning on eight hard DeepSWE tasks, with two runs per task.
Across all eight tasks, mean cost fell 19.0%, from $8.78 to $7.12 per repetition. The calibrated score was 2.48 points higher and passed a five-point noninferiority test (p-value=.032).
The 36.5% result came from the four tasks where the old instructions had caused the agent to continue or repeat reviews:
- Effect SSE HTTP API streaming
- Oxvg structural selector preservation
- Eicrud keyset-pagination cursor
- Yaegi Go embed directives
On those four tasks, mean cost fell from $5.83 to $3.70, or 36.5% (one-sided paired p-value=.034). The calibrated score was 6.48 points higher and passed the five-point noninferiority test (p-value=.037).
I also ran an ablation that removed only โrepeat any invalidated reviewโ without adding โDo not repeat plan review.โ That did not reproduce the result. On the same subset, cost fell only 19.5% (p-value=.177), and score noninferiority missed significance (p-value=.073).
The explicit stopping rule appears to be the important part.
This is why I'm strongly against automated self-improvement by agents without human review. Not because I'm against the concept. But because it just doesn't work.
A harmless-sounding instruction can trigger another plan review, which finds more work, which triggers more implementation, which triggers another code review. It can hurt performance and, just as importantly, increase costs. These kinds of things compound.
Write your skills yourself. Have your agent edit them, but you should review every single word.
I expect the latest cleanup of my own skills and instructions to reduce my token costs by roughly 3x, primarily by removing inappropriate instructions left over from earlier, less sophisticated agents. I'm planning to share the final results once I'm done optimizing.
So why have skills and instructions at all if frontier models are so good without them? Two reasons.
1. To encode the things I care about as a developer, many of which are truthfully just opinions.
2. To facilitate fully automated and unattended, long-running work.
I regularly run cross-agent loops using both Claude and Codex, across several reasoning levels, from the command line. They last hours and sometimes days. Skills make that possible. They just need to be very light.
Before Opus 5 and gpt-5.6 (Luna, Terra, and Sol) came out, my instructions and skills were roughly 10x their current size. Now my instructions are around 3,000 total tokens, and they seem to get smaller every day.
More to come soon about this benchmarking approach, which focuses on finding statistical significance with minimal costs. Follow me to stay tuned.
Useful links:
Anthropic, โThe new rules of context engineering for Claudeโ:
https://t.co/WglOBkDkVe
OpenAI, latest-model guidance:
https://t.co/OtOhXvw41z
OpenAI, reasoning-model prompting guidance:
https://t.co/9QYIH7QDNW
I'm obsessed with data, yet I had none to show if Agent Layer skills/instructions helped or hurt performance. I'm building a low-cost benchmark with @datacurve's DeepSWE tasks. I hope to share it soon. I need to reduce cost without hurting scores. https://t.co/kq3HD95Bus
Anyone using coding agents can create a personal assistant. I first built Steward to control access to my email. Claude and Codex have plugins for this, but Iโm REALLY glad I decided to build my own. Stew uses @LinkedIn, @X, @todoist, and more.
Read more: https://t.co/bxG0p3DMc1
Before posting more, I want to share how my personal agent Stew turns raw thoughts into X posts. I pick a topic or idea, type/dictate a free-form stream of consciousness. Stew uses my x-write skill (https://t.co/MkSRruXJdF) to write using my words. I edit and approve. Stew posts.
Stew, my personal agent, showed me a post from an AI newsletter framing agentic coding as the best video game ever. That really resonated. I'm lucky enough to play this game for work and fun. I'm going to start sharing what I learn, hoping it will help someone. Stay tuned!
Last night, after a weekend hackathon, I (soft) launched @iolify, a new desktop app console for AWS, initially focused on DynamoDB. More details coming soon.
https://t.co/OsDeCTYExB
Just upgraded, and noticed a couple features seemingly broken in @intellijidea 2022.1.4 on macOS:
1) File > Open > Scratch file does nothing
2) Help > Find Action... does nothing (Shift Shift does not work either)
Any tips on how to "auto-authenticate" callers in lower environments based on the presence of an `auth-bypass` header with @SpringSecurity OAuth2 resource server?
@datadoghq Getting flooded with alerts since yesterday. I found this issue which says resolved, but it looks like it's still an issue. https://t.co/wmAkWH3sr5