sandeepj

Co-Founder @CompuMatrice | We fix AI that fails in production
AI Observability · Evals · Agent Ops · Deployment -
Inc 5000 🏆 | Dallas 100 🏆

[sandeepj @sandeepj]

Model selection: U evaluated a pinned snapshot and you're serving an 'alias'. .. Provider ships a point release, your numbers now describe a model that isn't running. 'Pin it', re-run the suite before you move the pin, keep the old one warm a week. Cheap insurance :)

[sandeepj @sandeepj]

On Evals: Most prompt regression tests assert "the output looks right". That catches nothing. Assert invariants: required fields present, no invented tool names, refuses when context is missing, length in band.
... Otherwise a 2% pass drop tells you something broke, never what

[sandeepj @sandeepj]

@nutlope wow, $17 to train and $0/M output makes judge cost stop being a design constraint.... nice. The number I'd want before swapping: tev1-4B agreement with human labels vs a frontier judge on the same rubric. Cheap judging only helps if the agreement gap is small.

[sandeepj @sandeepj]

@neatlogs Good ep1 ... thanks. 'Trace reading' is the skill nobody teaches. What I'd want in the ep 2: how to sample traces when you have millions and the failure rate is 0.3%. Random sampling finds nothing. what I would like to know is What's your selection strategy?

[sandeepj @sandeepj]

Distilling the harness beat keeping it: 23.3% to 44.3%, above the 41.7% the base model scores with scaffolding attached. yes, worth sitting with if you maintain an agent framework ... well, some of your scaffolding is now overhead the model has outgrown.

[elvis @omarsar0]

Super interesting paper from Google and colleagues.

It studies where it's possible to distill an agent harness.

With the specialized harness removed, macro task success goes from 23.3% to 44.3%. That is higher than the 41.7% the base model reaches with the harness attached.

Harness-Zero uses the optimized harness only during training.

The optimized harness and the deployment harness have different action spaces, so a harnessing agent guided by the optimized harness corrects the student's responses in the deployment action space before they run. Those corrected runs become the training demonstrations.

Across 28 harness-induced behaviors in knowledge work, tool use and science, 82.3% are recovered on average. For frontier models using the same evolved harness, the agent-as-harness form also beats the code-as-harness form.

It remains to be seen how robust the approach is, but it's very interesting to see potential in harness distillation.

Paper: https://t.co/AiBm1p17Lp

Chat with Paper: https://t.co/Nda4XsZgrh

[sandeepj @sandeepj]

@omarsar0 The 38.5% of the benchmark reproducing the full score within 1.03 points is the number that matters if you run evals on every deploy.
Does the 200-question subset stay stable after a model swap, or do you have to re-derive it each time the agent changes?

[sandeepj @sandeepj]

@HamelHusain The row-picking framing is the useful part: most eval guides assume you already know which failure mode you have. Is there a row for teams whose problem is that the LLM judge drifts after a prompt change? That's the one I keep hitting in production.

[sandeepj @sandeepj]

FDE : The FDEs who get called back aren't the ones with the cleanest code. They're the ones who said the bad news early... A date slipping in week two is a scheduling problem. The same slip in week nine is a trust problem. ask me how i know :)

[sandeepj @sandeepj]

Agents are overwhelming us... wait till the Robots show up. - @Bencera vision.

[Ben Cera @Bencera]

I believe this is already manifesting as AI stops waiting for human instruction.

The next step is robots that can do the same in the physical world. Once we see that I believe we’re living through the final chapter of humanity as we know it.

[sandeepj @sandeepj]

Your fallback path has never run.
Provider rate-limits you at 2pm, and the 'cheap model' branch written in week one drops half the system prompt and returns a shape nothing downstream parses. Fire the degraded path on purpose, weekly, or "it isn't a fallback".

[sandeepj @sandeepj]

@simonw The useful reframe is latency budget, not capability. A decision call cheap enough to run on every trace changes what you can afford to monitor in prod: routing, guardrails, trace scoring. That is an 'observability unlock' before it is a modeling one.

[sandeepj @sandeepj]

What pushed the migration: eval ergonomics or the tracing? Logfire plus pydantic-evals keeps types and traces in one model. The part that bit me a little bit was porting existing judge prompts without silently shifting pass rates. Did yours hold across the move? Btw I was in fall 25 cohort of Shreya/Hamel eval course. The duo are awesome and killing it!

[sandeepj @sandeepj]

@braintrust Useful that you published where it lags, not just where it wins... Did you measure agreement with human labels per domain, or only against stronger judges? Groundedness holding while math and code degrade, points at "reasoning depth" rather than calibration.

[sandeepj @sandeepj]

@LangChain Going from 1% sampling to 100% coverage changes the failure mode — you stop missing rare regressions and start drowning in flags. Is there guidance on recalibrating judge thresholds against human labels before turning coverage all the way up?

[sandeepj @sandeepj]

FDE Hiring: a system design round tells you less than watching an engineer get told 'no' by a customer. Do they argue, go quiet, or ask what changed? Only the third one is doing the job. You can't interview for that, you can only watch for it :)

[Osmantic @OsmanticAI]

We’re gonna make sure Opensource AI wins and Local AI is the default 💪❤️

[sandeepj @sandeepj]

Agent observability: traces get written by the person who built the agent and read by whoever is paged at 2am, who didn't. They don't need a span tree, they need what "the model actually saw at the step that went wrong". If the trace can't show that, it's decoration

[sandeepj @sandeepj]

Clickhouse - deep dive. enjoy.
https://t.co/CANYDqnjLp

[sandeepj @sandeepj]

Benchmarking agents: You measured one request at a time, warm cache, clean input. Prod is 40 concurrent, cold, mid-conversation. The tail doesn't degrade gracefully, it stacks: queued tool calls, timeouts, retries on those timeouts. Bench at your "real concurrency" :)

[sandeepj @sandeepj]

@arjunnatarajan_ that's the part that matters. a retry loop three levels down just looks like root latency until you can query the span directly. cool.