Shaun Smith

[clem 🤗 @ClementDelangue]

Sad but still bullish on @nvidia!

I’m seeing a lot of people criticizing its structure and the lack of a clear CEO successor. Having seen it from up close, I think a lot of this criticism is ill-informed and overly simplistic (as often happens when a company is run differently from the norm).

Obviously, important to remember that this structure made NV as successful as it is and powered the AI revolution. But beyond its past success, I think it’s also what makes the company antifragile for the future.

In my opinion, NV's distributed leadership and comfort with chaos are much more of a strength than a weakness, even without Jensen. They give an unusual number of leaders room to step up and keep the whole machine moving (as we've seen in the past few days). They also keep the company flexible enough to adapt to the next paradigms (open-source/local/agents/robots/biology/chemistry/...). They also encourage hard-working, self-motivated team members with a strong sense of belonging and a willingness to help each other which will matter a lot.

Personally, with the right CEO (and NVIDIA has plenty of exceptional leaders internally who could take on that role), I believe NV can keep thriving and empower every company to become an AI company and every builder to become an AI builder.

To the @nvidia team: sending you strength and support, from me, the @huggingface team and so many people across the AI community. We need you and we’re rooting for you to carry @JensenHuang's mission forward! Taking care of his company, your company, is probably the best way to honor his legendary legacy!

💚💚💚

[Shaun Smith @evalstate]

I've been building with Monty since it launched - it's fantastic to work with. Sam is right here; high performance alternatives are needed. I'm also using it for PTC and it's seamless.

[Samuel Colvin @samuelcolvin]

🚀 Today we're launching Agent Sandboxes in @pydantic Logfire: a profoundly different kind of sandbox.

"The world doesn't need another sandbox" they say. Maybe. Or maybe the world doesn't need another firecracker/gVisor/VM infra-fest.

Our Agent Sandboxes are different. They're just an isolated interpreter with cold start times measured in milliseconds, where the code can only call the tools you give it and use the directories you mount.

A massive bonus is you can snapshot and resume a REPL weeks later, with the snapshots costing virtually nothing.

Programmatic tool calling, solved.

Start free, $1000 of free credit to try it in the first month 👇

https://t.co/yTjo0DAg0m

[Shaun Smith @evalstate]

❤️ stockholm

[Shaun Smith @evalstate]

How long has Arlanda station had a carpetted train platform? Weirdest vibe.

[Shaun Smith @evalstate]

@thdxr So many of the right things (like subtracting reward hacks from benchmark scores) work against you.
I report TTFT and measure TPS from stream start.

[Shaun Smith @evalstate]

LLMs are beautiful

[Shaun Smith @evalstate]

Receipt is the new absolutely right.

[Shaun Smith @evalstate]

Needed some [defensive] security advice on a repo.
Skip past Opus, Astra, Sol as can't be bothered to fight with their classifiers.
Select Kimi K3.1 (via Hugging Face inference / @togethercompute).
HOLY SHIT THAT'S FAST (and excellent result).

[Shaun Smith @evalstate]

@Poita_ What a lovely demonstration!

[Shaun Smith @evalstate]

@ElijahPopowitz How many trials were disqualified for reward hacking?

[Shaun Smith @evalstate]

With a stalwart like tb-21 you get a reasonable test of harness quality - once you baseline on that, these "mystery" benchmarks and charts that come out of nowhere with no reproducibility can be safely ignored.

[Shaun Smith @evalstate]

To save everyones time, I benchmark fast-agent, audit the traces and publish them publicly so you know you're getting pareto-optimal for your model. Comparisons are against equivalent published leaderboards when available.
We should normalise this, and kudos to GH-CLI team for doing similar.
https://t.co/dJdSGmHNsK

[Shaun Smith @evalstate]

Finally, there's no magic. All tool-loop based harnesses should be converging to similar score/efficiency. The main accuracy/efficiency losses come down to solvable defects exposed by the benchmark.
mini-swe-agent is a good baseline for current provider APIs.
If you are adding UX features that the model has in context, yes that will cost.

[Ryan Marten @ryan_marten]

Terminal-Bench 4.1 is here with updated verifiers for all tasks.

In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility

Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.

Leaderboard scores remain within previously reported confidence intervals.

[Shaun Smith @evalstate]

A huge upgrade, and a good place to handle plumbing away from the agent - a few words in a prompt and the whole thing "just works". Child panes are 👌.
Wondering now about custom panels for model-built widgets, charts, interactive displays (and whether that could work across machines...)

[Shaun Smith @evalstate]

@the_aiju I think they door dash it or something.

[Shaun Smith @evalstate]

@herdrdev Love it, added support in the default system prompts if started under HC https://t.co/nyT77Ae4FZ

[Shaun Smith @evalstate]

fast-agent 0.10.53 has a new variable in its default system prompt - if running under herdr-core it includes instructions for using other terminals and checking the hc --skill when doing so.

[Shaun Smith @evalstate]

@bnjmn_marie Oh 2 runs :( DeepSWE has very low repeatability (lots of within-job task flip). Please tell me they did their own baseline runs...?

[Shaun Smith @evalstate]

@AdamHoltererer Stream timeout adjustment often needed (I benchmark with extended periods at max).