Jack Klimov

Architect, ex-CTO. I put AI agents to work in engineering teams. Building Ushabti, an open-source board where agents work as teammates.

[Jack Klimov @jaklimoff]

@MaanavA51108 i remember in the CooperBench paper two agents splitting work did ~30% worse than one. letting them talk to each other didnt help. basically orchestration normally beats choreography 💪

[Jack Klimov @jaklimoff]

@Verdana86 same here! btw, have you found the ideal memory note size? i found out that in my case bigger notes make search less efecient

[Jack Klimov @jaklimoff]

I skipped embeddings in my agent's memory and kept a boring full-text index. A week later I read every search it ran.

57 searches. 11 came back empty. 2 were the kind of miss embeddings fix. The other 9 were plumbing.

Where the 11 broke:
- 5: the session wasn't indexed yet
- 2: it looked in notes, the answer was only in transcripts
- 1: the word only appeared in tool output
- 1: a database error on day one
- 2: a meaning miss, another language or a synonym

The best one: my agent searched memory for the model I'd wired in that same day and got nothing. The conversation hadn't been indexed yet. The memory couldn't remember what it had learned that day.

A stale index stays stale with vectors on top. A hook after every turn fixed it.

Then the fair test. My recall hook runs an LLM judge, Jev, that answers yes or no for each note the session may see. I ran bge-m3 against it on 20 hand-picked prompts. 13 want a note, 7 should show nothing.

- bge-m3, top 3: found a wanted note on 9 of 13, and showed 50 wrong notes
- bge-m3, a cutoff that keeps all 7 quiet: found 5
- Jev at 0.6: found 8, all 7 quiet, 6 wrong notes

As a ranker, bge-m3 is good. It just can't say no. Wanted notes scored 0.45 to 0.69, nothing-wanted 0.43 to 0.57. The ranges overlap, so every cutoff trades one failure for the other.

20 cases, one user. A smoke test, not a benchmark.

Claude Code and Cursor walked the same road at a much bigger scale: both dropped their vector index for grep.

So log your misses for two weeks before you buy a vector database, and tag each empty search.

"Other language" or "paraphrase" more than once a day? Buy it, and keep full-text next to it for exact identifiers.

"Not indexed yet" or "wrong store"? That's a cron problem. No embedding model has ever fixed a cron problem.

[Jack Klimov @jaklimoff]

@Verdana86 nope, I only tagged the empty ones. checking the hits for stale notes is a good next log! :) how do you handle it on your side, merge old notes or restructure them somehow?

[Jack Klimov @jaklimoff]

@Verdana86 agree. In my personal memory i found out that structured facts about me or my preferences tend to become rules and it is helpful to always feed them into context for certain projects. It saves an overhead of searching in a big memory bank.

[Jack Klimov @jaklimoff]

@alexhillman fair! 😈 the agent drafts, I clearly didn't edit that one enough. But it doesnt mean i don't write myself :) what gave it away?

[Jack Klimov @jaklimoff]

@GeoffreyHuntley Mine is looser: I set which projects each agent sees, but that's focus, not a wall. The hard line is the NATS login: the agent user can't write my records or the control bucket, so it can't speak as me. Where does cedar run for you, on the board or at each tool call?

[Jack Klimov @jaklimoff]

@agorokh Read it. 😎 My memory has the same split: rules the agent must follow load in full at every session start, and only facts go through recall. A rule the agent has to search for competes with the code in front of it, and your 1/8 shows who wins. Is the second model run next?

[Jack Klimov @jaklimoff]

@zachlloydtweets Same shift in my writing pipeline. Editor and fact checker had cleared my last post; a skeptic pass on another model found 10 problems after it went live, 1 high. So the rule now: four reviewers in parallel, merged into one review.md, and any open fix row blocks Approve.

[Jack Klimov @jaklimoff]

@sqs Two hard gates in mine. A push to main deploys, so the builder never pushes: I type it, and a pre-push hook runs the full check. And on my agent hub no agent can speak as me: a message from "human" counts only if the core recorded its hash first.

[Jack Klimov @jaklimoff]

@thorstenball On my solo projects it's gone. A task lands on main with no PR: a reviewer agent reads the diff, e2e tests gate the land. Next: a release train, where an architect agent wakes up now and then, checks the direction, steers. The hard part now: running hundreds of e2e flows fast.

[Jack Klimov @jaklimoff]

@emollick The form I run: agents are members, same standing as people, not a bot in the chat. Each one holds a card, reports on it, asks on it when stuck. Rights sit on the member, and an agent never gets admin. You follow one card, not the whole room.

[Jack Klimov @jaklimoff]

@levelsio @nosilverv @reasonmachines Mostly Claude Code here too. I went the other way on agents talking: mine don't message each other. One finishes, hands the card to the next role, the next one claims it. Nothing is sent; the whole record stays on the card. Fewer moving parts than a network, and I can read it.

[Jack Klimov @jaklimoff]

Want it? Run /statusline and paste this:

"Color the rate limits by pace: the share of the limit used divided by the share of the window gone by. Claude Code passes rate_limits.five_hour and seven_day, each with used_percentage and resets_at in epoch seconds; the windows are 5 hours and 7 days. Green under 0.8, yellow up to 1.2, red above. Count the first 10% of each window as gone by."

[Jack Klimov @jaklimoff]

My Claude Code status line said "50%" for the 5-hour limit. Fine or a problem?

Depends on the clock. 50% one hour in means I'm out by hour two. 50% four hours in means I'm fine.

So I color the limit by pace, not by percent.

[Jack Klimov @jaklimoff]

On a Pro or Max plan, Claude Code hands both limits and their reset times to the status line command. The whole rule is 10 lines, and Claude Code wrote it from one message.

0.8 and 1.2 were its first guess. I haven't touched them since.

[Jack Klimov @jaklimoff]

@agorokh So retrieval delivered the rule and the code in front of it won anyway. Found isn't the same as followed. Did you write that experiment up anywhere? I'd read it.