Arbit Chen

tech founder. ai agents, evals, shipping stuff. new dad in the bay

[Arbit Chen @itsarbit]

@cubertwang makes sense, a list is way easier to diff than 26k lines. my worry is the stuff nobody thought to put on it. do you write the failing cases by hand or let the agent propose them?

[Arbit Chen @itsarbit]

6.5M lines of Lean proof, ~26k lines of spec to trust.

DeepMind's Signal work had agents write proofs and humans sign off on specs. Lean can check the proof, but not whether you asked for the right thing.

How do you review 26k lines of spec?

[Arbit Chen @itsarbit]

@loneherd yeah the volume surprised me too. i'd guess most of it is the same few vendors bumping versions, so even a dumb per-vendor allowlist cuts the pile way down. post what you end up with

[Arbit Chen @itsarbit]

postman 12.33.4 quietly added a plugin, a startup hook and a remote MCP server to people's claude code and codex setups on upgrade. in codex it marked its own hook as trusted

if you use these at work, does your security team even know ~/.claude and ~/.codex exist?

[Arbit Chen @itsarbit]

@Dakshagarwal76 under 10 sounds about right. the part i keep going back and forth on is whether to prune the ones that never move or keep them as a regression floor. do you keep the dead ones around?

[Arbit Chen @itsarbit]

garry tan ran 7 agent setups on the same 20 coding tasks and 14 of the 20 scored the same in every one. so the whole comparison rode on 6 tasks

i'd bet a lot of internal eval sets look like this. if you run evals at work, how many of your tasks have ever flipped between two versions?

[Garry Tan @garrytan]

Out of curiosity I am doing some open source eval work on harnesses with frontier models and here’s what is coming back so far

[Arbit Chen @itsarbit]

@rohanpaul_ai one of them got 5.2 points more reliable and lost 10 tasks it used to solve, and the average just says it got better. a per-task diff against the starting version should be the default report for this stuff, not an appendix

[Arbit Chen @itsarbit]

@omarsar0 owning them is the easy part to say. keeping them current is where most sets quietly die, the tasks you wrote at launch stop catching anything once the product moves. who'd you put on the hook for that, eng or the domain folks?

[Arbit Chen @itsarbit]

@thorstenball you can already see it in hiring. a take-home that's just "implement this spec" says almost nothing now, an agent can knock it out before lunch. asking someone what they'd cut from the spec and why tells you way more

[Arbit Chen @itsarbit]

@karimhijazi alerting on anything that marked itself trusted is the bit i'd steal. legit tools basically never need to do that. have those teams caught anything real with it yet or is it mostly noise so far?

[Arbit Chen @itsarbit]

@GergelyOrosz the other half of that clip is tibo saying they work in public channels and open docs up broadly on purpose so codex can read them. at most companies a lot of the useful context lives in DMs and locked docs. how do they handle the stuff that has to stay private, like HR or comp?

[Arbit Chen @itsarbit]

@loneherd hashing at session start is cheap enough to just do. the annoying part is plugins that legitimately update themselves, you'd get a diff every week. would you allowlist by vendor or review every change?

[Arbit Chen @itsarbit]

update: postman said on the github issue today they're reverting it. reverting doesn't uninstall anything tho. if you upgraded since oct 24 the plugin, the hook and its trusted hash are probably still sitting there. `npx @postman/postman-plugin status` shows what got added

[Arbit Chen @itsarbit]

@rohanpaul_ai @livingbrain 22.6x fewer tokens is the headline but the hard part is retrieval grabbing the right few memories. i'd want to see that 92.2% LoCoMo number on a store that's a year old and full of stale stuff

[Arbit Chen @itsarbit]

@loneherd reading it before it's trusted is the right default. does your guard also catch stuff written into those dirs between sessions, or only what fires on tool calls?

[Arbit Chen @itsarbit]

@PineapplesDev yeah that was the part that got me. the plugin and the MCP server you can at least spot later, but a hook that approves itself skips the one check you had

[Arbit Chen @itsarbit]

@karimhijazi yeah diffing after updates is probably the only real check right now. feels like ~/.claude and ~/.codex belong on the same watch list as launch agents and shell rc files. have any of the teams you talk to started doing that?

[Arbit Chen @itsarbit]

@milan_milanovic checks is the layer people skip. a hook that runs the tests before the agent is allowed to say done does more than another page of CLAUDE.md rules, and it's a tiny bit of config

[Arbit Chen @itsarbit]

@mattpocockuk the 100% coverage part is carrying that example. most gnarly perf modules have a couple of tests at best, and that's right where agents get confident and wrong. agree on the direction, the tests just have to show up first

[Arbit Chen @itsarbit]

@DanKornas error explorer is the part i'd actually open. agents rerunning the same failing command over and over eats a lot of a long session. does it group retries that only differ by a flag or a path?