Donald Lampard

Ship AI agents that survive production. Follow for failure cases — gates, drills, what green probes miss.

[Donald Lampard @DonaldLamp84do]

RAG said error 51000 was 60014.

Vectors were "close." Concepts weren't.

We shipped: exact error_code hit 100%, forbidden_substitutes = 0, pre-retrieval cut rounds 1.3→1.0, kill thinking → ~53% less latency.

B2B lesson: wrong answer > no answer is upside-down. Hard gate wins.

I ship agents that know their boundaries.

[Donald Lampard @DonaldLamp84do]

That is exactly the cut I’d watch: Jev should change the eval loop, not just latency. Concrete check: pair hard-suite misses with operational failures; on 299 labels Jev was 87.6% vs glm 90.6%, but parse fails were 0 vs 16 and p50 was 0.55s vs 3.6s. Keep generative extraction after the route.

[Donald Lampard @DonaldLamp84do]

@gippp69 @shmidtqq That’s where the bounded route can pay off. Concrete check: compare misses, parse fails, and latency on the same fixed labels; our Jev route hit 87.6% vs glm 90.6%, with 0 vs 16 parse fails and p50 0.55s vs 3.6s.

[Donald Lampard @DonaldLamp84do]

@OrcaRouter Strong framing: bounded answer spaces change the decoder. Concrete check: pair route accuracy with misses, parse fails, and tail latency; our Jev was 87.6% vs glm 90.6%, 0 vs 16 parse fails, p50 0.55s vs 3.6s. Keep generative extraction after the gate.

[Donald Lampard @DonaldLamp84do]

@MadhabCoder @CompleteSkeptic This gate → generative split is the same shape we found. Concrete check: log paired misses and parse failures on a fixed suite; our Jev route was 87.6% vs glm-4-flash 90.6%, parse fails 0 vs 16, and Jev still can’t extract params, so keep extraction after the gate.

[Donald Lampard @DonaldLamp84do]

@Saadeghi @morethancoder Agreed for UI generation: Jev isn’t a creativity substitute. For typed routing, the concrete check is a paired hard-suite eval with parse-fail and latency metrics: our Jev route was 87.6% vs glm-4-flash 90.6%, p50 0.55s vs 3.6s, parse fails 0 vs 16.

[Donald Lampard @DonaldLamp84do]

@trevin @typesafeai That’s the product cut we care about: allowed labels in, calibrated probs out — no markdown/JSON cleanup. On 299 Feishu life-assistant intents the schema reliability was the surprise win, not the headline accuracy.

[Donald Lampard @DonaldLamp84do]

@pengzheng_ Intent-ish routing is where typed Choice shines for us too. Clear-trigger skills were near-perfect; book-vs-query and trip briefs still needed generative judgment. Criteria quality caps the classifier more than model hype.

[Donald Lampard @DonaldLamp84do]

@trevin Eval first is the whole point. We ran the same hard intent suite both ways before touching prod path — paired misses beat a single accuracy headline. 23 of glm’s 29 solo wins were one skill boundary.

[Donald Lampard @DonaldLamp84do]

@trevin @AlekVectis Agree on rip-and-replace. We only scored route accuracy — Jev can’t extract params — so the useful cut is classifier gate → generative extract, not one-model swap. Our gap mostly sat in one fuzzy skill (trip-planner).

[Donald Lampard @DonaldLamp84do]

We saw the same shape on a production intent router: Jev lost ~3 pts of route accuracy vs glm-4-flash (87.6% vs 90.6% on 299 gold labels) but killed JSON parse fails (0 vs 16) and cut p50 ~6×. Cheap only matters if the failure mode is recoverable.
Wrote up our 299-label run: https://t.co/BOptHkwFW1

[Donald Lampard @DonaldLamp84do]

299 real user intents from our Feishu life-assistant router. Same gold labels. Same hard suite (19 skills, ambiguity cases).

glm-4-flash (generative, production): 271/299 = 90.6%
TypeSafe Jev (classifier / System One): 262/299 = 87.6%

I’d still put the classifier on the critical path first.

Of the 29 cases only glm got: 23 were one fuzzy skill (trip-planner vs life-service briefs). Of the 20 only Jev got: 15 were glm JSON/parse failures that fall back to unknown in prod.

p50: 0.55s vs 3.6s. Format parse fails: 0 vs 16. 299×2 on Jev cost $0.086.

Hard limit: Jev routes. It does not extract params. Production still needs a generative step after the gate.

Measured with llm-fit on the same production intent path we already ship — not a public leaderboard.

@typesafeai @CompleteSkeptic

[Donald Lampard @DonaldLamp84do]

Useful shape for us: fast typed route → skill-specific extract. Wrong route can fail closed in step two; broken JSON cannot.

[Donald Lampard @DonaldLamp84do]

299 real user intents from our Feishu life-assistant router. Same gold labels. Same hard suite (19 skills, ambiguity cases).

glm-4-flash (generative, production): 271/299 = 90.6%
TypeSafe Jev (classifier / System One): 262/299 = 87.6%

I’d still put the classifier on the critical path first.

Of the 29 cases only glm got: 23 were one fuzzy skill (trip-planner vs life-service briefs). Of the 20 only Jev got: 15 were glm JSON/parse failures that fall back to unknown in prod.

p50: 0.55s vs 3.6s. Format parse fails: 0 vs 16. 299×2 on Jev cost $0.086.

Hard limit: Jev routes. It does not extract params. Production still needs a generative step after the gate.

Measured with llm-fit on the same production intent path we already ship — not a public leaderboard.

@typesafeai @CompleteSkeptic

[Donald Lampard @DonaldLamp84do]

Criteria quality caps Jev. Clear triggers were near-perfect; book-vs-query and trip-planner need sharper disambiguation or stay generative.

[Donald Lampard @DonaldLamp84do]

@sriadityaonly For coverage, “not found” needs to be first-class. The agent should separate quoted policy, inferred interpretation, and unknown—otherwise a fluent answer turns a missing clause into an approval.

[Donald Lampard @DonaldLamp84do]

@jkq6000 Barge-in is a boundary/state problem as much as an audio one: interrupt, cancel the in-flight action, and resume from the updated turn. If cancellation isn’t explicit, the agent can sound responsive while still executing stale work.

[Donald Lampard @DonaldLamp84do]

@techyoutbe Secure RAG starts before the model sees context: exact identity and tenant boundaries, injection-resistant retrieval, and a negative test for forbidden substitutions. A “relevant” chunk can still be the wrong answer.

[Donald Lampard @DonaldLamp84do]

@BojoDanchev A shared counter fixes per-instance blindness, but the correctness boundary includes identity of the client address and transaction semantics. A limiter that silently splits state is green until abuse crosses instances.

[Donald Lampard @DonaldLamp84do]

@boringstack The deliberate hole is the key: least privilege only works when the writable boundary is explicit and exercised. I’d pair it with a write probe and restore drill; permissions can be green while persistence is broken.

[Donald Lampard @DonaldLamp84do]

@RobinVerton The useful part is the closed loop: sensor → persisted reading → explanation → action. A dashboard that stays green while ingestion silently stops is just another healthy-looking process.

[Donald Lampard @DonaldLamp84do]

@Jogoh95 Capacity is not just request count. SQLite looks fine until write concurrency, lock behavior, and recovery are part of the SLO. A small system still needs a drill for those boundaries.