@BoboLinNotes Token budgets deserve more attention. A hard cap on tool calls per task plus an "I don't know" exit stops most runaway loops. Where would you put that cap in the learning order, before or after context engineering?
@rvaniaaaa Context rot often shows up as the agent quietly dropping a constraint it read 30k tokens earlier. Does the research show where in the window those instructions get lost first?
@e_opore Checkpointing is the easy part. The hard part is making tool calls idempotent so a restored agent doesn't resend the email or rewrite the row it already handled. Do you log a step id for each side effect before resuming?
Treat every "should" in a standard as a "shall" and your compliance tool will hand back hundreds of flags.
Standards are drafted on purpose. "Shall" is a requirement. "Should" is a recommendation. Read them the same way and every recommendation becomes a finding.
So engineers learn to skim the flags. And the one that is an actual requirement gets buried in the pile.
A flag list nobody trusts is worse than no list.
At CertBuddyAI, our agents answer only from the specs, standards and reports you give them, and a "should" doesn't come back dressed up as a violation. Every flag points to the section and page, so an engineer can check it in a minute.
Engineers still make the call.
@perperp_er If the reward pays for fewer tools, what stops it from learning to skip the one call that actually catches bad patches, like re-running the tests?
A test report can show PASS on every test and still violate the requirement.
It happens when the tests were written to the wrong reading of the clause. Each test passes, and each one checks the wrong thing.
Nothing on the program compares the report's verdict against the requirement text. So the mismatch makes it all the way to the authority's comments.
I made the same argument about agent evals this week. A patch that passes the tests but breaks a stated rule should score zero. Otherwise the eval quietly rewards clever rule-breaking. A test report works the same way: PASS against the wrong reading is a fail.
At CertBuddyAI, our agents read only the specs, standards and reports you give them. They flag every place a verdict and a clause disagree, with the line in the report and the clause it conflicts with, down to section and page.
Engineers still make the call.
@EmmerichMarcelo I'd score these all-or-nothing: if a patch passes the tests but breaks a stated rule, it gets zero, otherwise the eval quietly rewards clever rule-breaking. How do you write the review rules so a grader can check them, as plain rules or with a judge model?
At every certification gate, someone senior puts the test report on one screen and the spec on the other and reads both, line by line.
It has to be them. They're the ones who know which mismatches matter and which are just wording. So they do it at Design, at Test, and again before Cert, every time either document is revised.
Hours of the most experienced person on the program, spent finding where two documents disagree.
At CertBuddyAI, we're building agents that do that first pass, using only the specs, standards and reports you give them. Each mismatch comes back with the line in the report and the clause it conflicts with, so checking one takes a minute.
The senior engineer still decides which ones matter. They just stop being the diff tool.
Two teams can read the same regulation and come away with two different requirements.
Systems reads the clause one way. Test reads it another. Each spent weeks searching CFR, EASA and ISO by hand, and each wrote its reading into its own documents.
Nobody compares them, because nobody knows there are two.
The gap shows up at the design review. Or worse, in the authority's comments.
At CertBuddyAI, we're building agents that answer only from the regulations and standards you give them, never the open web. Every team gets the same answer, pinned to a section and page. If someone disagrees, the clause is right there to argue with.
Engineers still make the call.
The amendment that hurts is the one that publishes after your submission is already in.
Your design was traced to the rules as they read on day one. Then a regulation or standard changes mid-program.
Nobody on the team is rereading CFR or EASA that month. So the change turns up in the authority's comments.
The rework lands on the ship date.
At CertBuddyAI, we're building an agent for exactly that window. It flags the changes that affect your product while there's still time to act, and every flag points to the section and page that changed.
Engineers still decide what to do about it.
@victorialslocum Per-agent identity answers "which agent spent it," but when an agent spawns sub-agents mid-run, do they get their own identity or inherit the parent's spend cap?
The models that searched the most scored the worst.
Deepnote put 12 frontier models in a fake company's Slack and asked one question: help, or stay quiet?
The two making ~35 tool calls per task finished last. One hit the time limit without sending anything in 27 of 50 runs.
Opus 5.5 made ~10 calls and scored 7.84.
More tool calls isn't diligence. It's an agent that can't decide.
Cap the search. Force a decision: a message, or silence.
@cyber_rekk The scary part is that the evals were what gave the false confidence. Is anyone checking what the agent actually executes at runtime instead of only scoring its task outputs?