Building software since 2001. Zscaler, Node.js in 2010, founded two companies. Now: authorization and governance for AI agents. Also math, guitar and farming.
In 2002, as a trainee, I wrote C and assembly for the motherboards monitoring the nuclear reactor at BARC. Hundreds of them on a shared Ethernet bus, all shouting status at each other.
We designed systems to be subordinates. They told you what was happening, and somebody above them decided what to do about it.
Twenty-four years on I build AI agents, and the question hasn't changed. What is the machine allowed to decide, and who answers when it's wrong.
I write about that here. Also math, guitar, and why those turn out to be the same thing.
An agent fleet scales because nothing on the hot path has to agree with anything else.
That is also why revoking one agent's access never reaches it.
What "immediately" has to mean, and the three questions any containment bound has to answer.
@appenz@YODELAGARZA The certifi detail is the tell. Most inventories answer which packages you installed, and the question during one of these is which processes hold a copy of the trust decision.
Agree, and the part that surprises people is that the binding constraint is usually not throughput, it is what you are obliged to do inside the latency budget. We ran policy evaluation inline in the proxy path at Zscaler, and every feature request was really a question about what fits before the user notices. Inference serving is walking into that same conversation.
Calibration is a property of a distribution, which probably explains the split. WildGuard and When2Call look like what these models were fit on. HLE is selected for difficulty, so it sits outside that by construction, and a confidence mapping learned elsewhere will overshoot there.
@JoshARosen Scoring the trace doesn't tell you the agent got stuck. It tells you the evaluator thinks so. And honestly, unless somebody checked that model against human raters, you don't know what that's worth.
Locking the manifest and tuning until the suite hits 94 is how a test set quietly becomes a training set. We built the same thing for fraud rules, replaying every change against frozen historical traffic.
The part that bites is that the policy starts fitting the window rather than the problem, while the suite score keeps going up.
Reviewer disagreement has at least three sources and they want different fixes. Rater noise averages out if you add raters.
A genuinely borderline case stays borderline however many you add. And disagreement because nobody ever decided the policy produces a model that splits the difference and satisfies neither.
btw I don't know a cheap way to tell the second from the third, which is the part that actually bites. Does the roadmap treat them separately, or does low agreement come out as one number?
80% agreement with SDR edits against 38% for the judges is a real gap. What I keep wanting next to it is how often two SDRs edit the same draft the same way.
Without that you can't tell whether the 80% is the model learning a shared standard or learning the habits of whoever happened to edit that example.
btw, we never measured this properly either, so less a criticism than the thing I wish we'd had.
The leverage there probably wasn't the regulation, it was that you could supply a reason. Reg E lives in the policy binder and nowhere on the frontline screen, so the agent sees a decision and a queue and no field for why.
A VP has that field. I built fraud tooling that fed those queues, and when we later swapped a rule engine for a model we made it worse, because the rules at least emitted a reason code you could argue with.
Worth separating those further. We swapped a rule engine for a model and it caught more fraud at a worse net, because review cost per case went up rather than down. A rule tells the reviewer what fired. A score tells them only that something is worth looking at, so they rebuild the case from scratch every time.
Built a Trie over the full US Census TIGER database in 2002 for LA taxi dispatch autocomplete, ~30K street names, so the branching-node trick is a familiar one.
Footnote 5 is the interesting one. You measured trie_tree against trie_decode and found them equal, and measured speed against vLLM.
Did you ever get accuracy against the unconstrained baseline?
@HamelHusain The eval panel is the part i would like to know more about. How often do Amara and Jon actually agree on the same case? Whatever that number is, it sets the cap on what the evaluator can learn from them. eh?