Does anyone know a benchmark where coding agents build the same service in different programming languages, and someone counts the bugs and the tokens?
I need one because I'm building a programming language. It's called Borsuk🦡, after the mathematician Karol Borsuk. It also means badger in Polish.
Agents write most of my code now. But writing was only about a fifth of an engineer's week, and with agents it drops to a few percent. Review, checking and understanding stay. That's why I think agents cap most teams near 2× today. The number that matters most is how often the agent is wrong.
To get to 10×, checks have to stop about 99% of bad changes before a person looks at them. Today they stop maybe 15%. Every change should come with a proof that a person can check fast.
Borsuk is built for that part of the job. Agents write it. People read it through tools, as tables, diagrams and traces. Every program is a state machine with typed effects, so every state change and every side effect is explicit. Every business rule is a law, and the tools check it and tell you how strong the evidence is. Every release gets a behavior diff against the last one. Crashes, retries and lost replies are the runtime's problem, and the runtime has its own proofs and tests. There is one way to write each thing, so an agent has very little room to be wrong.
I want it for services of every size. My blog, your pet website, a shop, a booking system, a payment service. A small app runs in memory or on one disk. A big one runs on many nodes with the same code.
👀 The closest thing I've seen is Bend 2 by Victor Taelin. It also puts laws and proofs in front of AI-written code, and it's fast. Borsuk goes after a different job. It's for services that keep state, so it deals with crashes, retries, lost replies, upgrades and many nodes. The tools also try to find each proof on their own first, with SMT and friends, before an agent writes one by hand in Lean.
I want to write end-to-end checks and tests for the critical logic, and nothing else. The compiler, the checker and the runtime cover the rest. ⚡ The speed target is hand-written Rust.
If you designed a programming language assuming coding agents would write most of the code, what would you make the compiler refuse to accept?
I’m sketching one called Borsuk 🦡 for high-throughput distributed systems. My five rules so far:
1. Strong types all the way to the boundary. An ID, a reply, and the right to take the next action are distinct types.
2. Explicit state machines. You can read a transition and see the input, next state, and effects it requests.
3. Private modules by default. An agent changing checkout should be able to use payments correctly after reading its public contract.
4. Ownership permits fast in-place mutation. A hot path can promise zero heap allocations, including when the queue is full.
5. Specs are part of the language. A module can promise “once I acknowledge this write, it survives a crash.”
The tools I’d want alongside it: searchable, version-matched docs over MCP, task-oriented prompts, and the same verifier in the CLI, MCP, and CI. Ask “what breaks if I move this acknowledgment one line earlier?” and get a replayable crash trace. The output should say exactly what was proved and what was only tested.
What’s a bug your coding agent wrote that sailed through CI? I’d love to hear what rule or tool would’ve caught it.
Agents write code ~10× faster. So where are our 10× engineers?
I built a simple model to see why, and recorded a 20-minute talk about it. Speed up that only affects code generation (even with "perfect" code) and the ceiling is ~2×.
10× needs fewer decisions per change.
@mattpocockuk@_lopopolo@nateberkopec Basically the idea is to give full elasticity to the agentic system that deliver the outcome. Then you eval against the outcome (plus probably costs and couple other of economical units). This way the system in theory can self improve and adapt as needed.
my daughter turned five! to hide from some party duties I took a very long shower.
I came out with a thesis and posted it at work. 17 replies in one hour.
the thesis: we should build systems that build our product. not the product directly. (scary, I know)
and the metric: how many human touches does one PR need, while the bar for bugs and regressions stays high.
the best pushback I got:
- trust needs responsibility, and you cannot give an LLM consequences
- you cannot verify a model with the same model stack
- compilers are deterministic, LLMs are not, so "nobody reads machine code" is a weak analogy
- go tactical first, invest in people and culture first
all fair. but nobody proposes "no oversight", and I do not either. you stay responsible for the system and its effect in the very same way.
I just want to move the human from the line of code to the system that wrote it.
please tell me why this is a bad idea. I mean it. I will not defend this ground for now, but I will keep bugging 🙂
I can recommend doing this, my process is still quite rudimentary, but I went from "ai-slop-grenade-generator" to "approved" w/o comments in less then a week (and mining past prs for comments to extract style). I've used expert guidance though - the domain was completely new to me (react native). It's still plug-and-play on multiple repos, BUT - using linting and all of these static checks very aggressively is way forward. (e.g. @poteto is recommending that for distributed systems with formal verification)
I think the meta-rule we are observing is making bad states illegal at zero-token-cost, so agents cannot make mistakes.
before I ask a human to review my PR, I make the bot happy first.
attention is our most valuable resource. I will not spend yours on stuff a bot can find.
so I made a quite simple skill, a copilot loop:
1. ask copilot to review
2. one subagent per round fixes the comments
3. push
4. repeat until copilot and the static checks are clean
5. only then ping a human
one fix agent per round, not per file. the comments of one review often touch the same files, and two agents in one file is how you get a very creative merge.
this month copilot reviewed 82 % of my PRs. 437 reviews. most replies were "good catch, fixed".
my favourite line from a PR description: "(claude wrote it, and Copilot reviewed it until it had nothing left)"
so robots review robots first, and humans get what is left. you can just tell your claude to steal this skill 🙂
over the weekend I read and built a lot around AgEx.
agentic experience. (cool name, I know)
we spent years on DX. fast builds, nice errors, docs that nobody reads.
and now the main developer in my repos is an agent. and its experience is honestly terrible:
- native build + simulator boot before it sees one screen
- it taps and scrolls like a person, 1 to 3 min per screen
- the taps flake
- it waits 45 s for a button that is optional
- it squints at screenshots to guess if a layout is right (THERE IS THIS BUTTON WHAT DO I DO?)
one small UI change this month: the code took under 15 min. the verification took about 120.
the agent is not slow. our tools are slow for it.
so basically AgEx is DX for a developer that never sleeps, never complains and pays by the token. I think we should spend way more time on this - and I will keep bugging people about it.
I spent two weeks making mutation testing fast (and a crapload of tokens out of my own pocket).
then I deleted all of it. 801,198 lines (yes, I counted - it is the metric I am most proud of this month).
day 1: I merge a fork of a mutation tool into my AI review setup. 743k lines. warm sandboxes, parallel seats, shared build cache. I feel like a genius.
day 12: I finally check what it found.
- 89 to 97 % of every review round was mutation
- 102 unique findings
- 3 real bugs - and another reviewer found all 3 anyway
- 0 bugs that only mutation found
so basically I built a very expensive space heater for my laptop.
the replacement is boring: show me a test that fails when you revert the fix. the old rule proved 25 of the last 60 rounds. the new one proved 49.
building it was free. keeping it was not - I learn this again every month, because I am a slow learner 🙂
@nark3d > Automating a judgement you haven't settled just makes the disagreement faster.
and that's good thing. conflict is not always bad
> Disagreement sits in "pick rules aggressively" and "every decisive PR comment leaves something behind".
I see here just a shorter feedback loop
The more code agents can write, the more valuable boring engineering becomes.
Linters.
Types.
Tests.
Consistent domain models.
I think these are some of the highest-return investments we can make in agentic coding.
Bad practices in a codebase are examples an agent can copy.
Historical reasons, shortcuts, w/e. They're still there.
Add checks that stop new occurrences. Start fixing the old ones.
Agents can help write those checks, too.
Every decisive PR comment should leave something behind.
A lint rule.
A regression test.
A type constraint.
An explicit domain rule.
We already paid the cost of figuring something out.
Let's stop paying to rediscover it.
If a rule can be checked deterministically, why keep spending tokens asking an agent to reason about it?
Write the check.
Run it inside the agent's loop.
Give it enough feedback to fix the failure.
Catch it before a human has to.
5/ We should invest much more in domain understanding and modelling.
What can exist?
Which transitions are valid?
What must always be true?
Then encode those answers.
Make illegal states impossible to represent, as much as we can.
Give the agent less room to guess.
An agent will either thrive or get drunk on the quality of your codebase.
Consistent concepts give it a useful pattern to follow.
Five abstractions for the same thing give it five ways to make a mess.
We should care a lot about what we're asking it to copy.
Every repo should have one code style.
Code shouldn't be a way to express yourself.
Pick rules aggressively. Automate them. Iterate when they're wrong.
We don't need to design every convention by committee.
It's easy to change a rule.
Agent capability keeps growing.
Human attention is still very finite.
We need better ways to review changes:
Show me the domain model.
Show me the flow.
Show me which assumptions changed.
Help me understand what I'm approving.
What I want from this:
Less time thinking about code mechanics.
More people able to ship a small feature.
More confidence in agentic workflows.
Automate the rules we've already agreed on.
Spend our attention on whether we're building the right thing.
Then extend this across repos.
Shared concepts.
Explicit contracts.
Checks that catch incompatible changes.
An agent working in one repo needs to understand the assumptions it might break in another.
The boundary between repos is still part of the system.
All of this is very software-engineering oriented.
But I keep wondering:
What would it take to 10x our science results?
Or how quickly we deliver them?
Better coding agents should make us more ambitious about the problems we choose to work on.
Pick a measurement process or a piece of hardware.
Have we tried systematic hill climbing?
Explored approaches beyond the current design?
Built a loop that measures whether each change actually helps?
Could it be 10x better, cheaper, or faster?
Let's find out.