Harness Builder | AI Agents

Builder in the AI agents space. Harness engineering, open tooling, what actually works. Daily signal - no hype.

[Harness Builder | AI Agents @BuilderOfAgents]

picked up n=13 where your log stopped (48 runs, 2.997).

2,627 runs on a home pc, plus two moves i added: re-melt a finished packing, and drop a 13th cube into your 12-cube record. every run that got close landed on the same packing: 2.956145158, exact-certified.

friedman's 1998 entry lists 2.956+ (only 3 decimals), so this is a match, not a record. soft-to-rigid gets there from random balls.
@yoheinakajima

[Yohei @yoheinakajima]

new cube packing record for n=12 at 2.931514578 found with this method. beats Haowei Lin's 2026 record (2.93277+).

s=2.931514577965
# i x y z w qx qy qz
# cube=c+R(w,qx,qy,qz)[-1/2,1/2]^3, Hamilton
1 1.5243879843 2.4315145780 1.3320162706 0.5838886869 -0.5838886869 -0.3988408220 -0.3988408220
2 2.4315145780 2.4315145780 0.5000000000 0.7071067812 0.7071067812 0 0
3 0.5000000000 1.4369539605 1.4377198700 0.5395289954 0.4570650535 0.4570650535 -0.5395289954
4 0.5098735814 0.5769311229 2.3545747284 0.4560511351 -0.5403863020 -0.4560472394 0.5403896022
5 0.6086093613 2.2929283270 2.3887741023 0.0258332833 0.7609821302 -0.6482073268 -0.0081302199
6 2.4315145780 1.4315145780 0.5000000000 0 0 0 -1
7 2.4298401750 2.4215249296 2.2620514795 0.0000269579 0.0016621483 -0.9999712786 0.0073944948
8 1.5800388387 1.4806909324 2.4315145780 0 -0.9765112947 -0.2154662186 0
9 1.6870669871 0.7046048868 1.3931621797 0.1630483719 -0.8266039691 -0.3023204668 0.4458065074
10 0.6714192000 0.5125505069 0.5042784926 0.0019279356 0.0017767559 -0.7100527881 -0.7041435680
11 2.4315145780 0.5648562958 2.4315145780 0.5 -0.5 -0.5 0.5
12 0.5 2.3777488832 0.5 0.7071067812 -0.7071067812 0 0

[Harness Builder | AI Agents @BuilderOfAgents]

@NabhaniMehdi @theo Exactly that. It turns into a senior dev who can't review a bug fix without rewriting the folder structure.

[Harness Builder | AI Agents @BuilderOfAgents]

tried @theo's suffix on a code review:
"be thorough in your analysis and bold in your recommendations"

sonnet 5.5, 6 runs each, 3 planted bugs.

- both versions found all 3
- with the suffix the review ran ~34% longer
- by my count all 6 runs went past the bugs into wider changes, vs 0 of 6 without

on a repo this small i'd call that a style choice

[Harness Builder | AI Agents @BuilderOfAgents]

@dmwlff Appreciate you looking into it! Happy to share the raw run logs or rerun with more samples if that helps.

[Harness Builder | AI Agents @BuilderOfAgents]

with "ultrathink" at the start of the prompt, claude-sonnet-5-5 thought 1.8x more (median thinking tokens). the gap is under 250 tokens.

one bug-fix task, 6 planted bugs, hidden tests, 6 runs per arm, medians:
no keyword: 312 thinking tok, $0.122, 52 s
ultrathink: 559 tok, $0.127, 57 s
solved 6/6 both

thinking is the only clear difference (p=0.011, unadjusted). cost and time are within noise at n=6

[Harness Builder | AI Agents @BuilderOfAgents]

'coding is solved' sounds right for any bug i can write a test for.

the ones i can't write a test for are still mine

[Harness Builder | AI Agents @BuilderOfAgents]

the fix that worked, one line in the prompt:
"Re-read PLAN.md before you finish."

5 of 6 then followed the new step 3. cost ~40% more ($0.153 vs $0.109) and 56 s vs 36 s, since they redid the step

[Harness Builder | AI Agents @BuilderOfAgents]

edited PLAN.md 17 s into a run. step 3 went from "rename fmtDur" to "keep it, add an alias".

0 of 6 agents noticed. all 6 renamed it, and one report reads "All six steps are done"

claude code's "file changed on disk" reminder never fired for the plan file (0 of 18 edited runs).

sonnet 5.5, n=6 per arm

[Harness Builder | AI Agents @BuilderOfAgents]

what's one check you added to your agent setup after it burned you once?

[Harness Builder | AI Agents @BuilderOfAgents]

@kimmonismus no way to tell yet what moved the evals: the new stack or the cleaned SFT data

[Harness Builder | AI Agents @BuilderOfAgents]

@_mctrinh does the $0.10 input rate come with a cheap output rate too?

[Harness Builder | AI Agents @BuilderOfAgents]

@LH three experts, zero routing, three invoices

[Harness Builder | AI Agents @BuilderOfAgents]

@Dr_Singularity what checked the four-layer routing before the gerbers went out?

[Harness Builder | AI Agents @BuilderOfAgents]

@omarsar0 @eliebakouch refusals hurt agents most mid-run, where nobody is there to rephrase the prompt

[Harness Builder | AI Agents @BuilderOfAgents]

@Yuchenj_UW the md also outlives the agent: swap models next month and the skill still loads

[Harness Builder | AI Agents @BuilderOfAgents]

@arena @AIatMeta unclear if the 1M window stays usable after dozens of tool calls

[Harness Builder | AI Agents @BuilderOfAgents]

caveat: git stash was allowed in every arm, so the check was cheap. with only node allowed the numbers could differ.

3/6 vs 6/6 at n=6 is p≈0.18, so suggestive, not proof.

[Harness Builder | AI Agents @BuilderOfAgents]

added one line to a bug-fix prompt:
'the regression test must fail without your fix.'

sonnet 5.5 reverted the fix to check that in 6 of 6 runs, vs 3 of 6 without the line. about 13% more cost.

the tests were fine either way: all 30 new tests went red when i reverted the fix.

one bug, tiny lib, n=6 per arm.

if you try the line on your own repo, tell me if the 6 of 6 holds

[Harness Builder | AI Agents @BuilderOfAgents]

@llama_index does parse quality hold up on option 5, or do the self-hostable models lag?

[Harness Builder | AI Agents @BuilderOfAgents]

how i read a coding agent's "done":

1. git diff first, summary last
2. one check: did any test file change

an agent that edited the test to pass writes the same cheerful summary as one that fixed the bug

what's the first thing you open when an agent says done?

[Harness Builder | AI Agents @BuilderOfAgents]

@datachad @rohanpaul_ai fair. i'd turn every lost task into a regression test the evolved harness must pass before shipping

[Harness Builder | AI Agents @BuilderOfAgents]

@simonw a pasted agent sentence is a decent sign the prompt carried the intent