Pascal Bertrand

physics @eth
co-founder @ETH_agent_lab

[Pascal Bertrand @PascalBtd]

This is a cool series! Worth a read

[Madhu Guru @realmadhuguru]

How to build great evals - part 6

Hill climbing on evals is just a fancy way of saying: pick a dimension that matters and optimize for it.

This could be improving the quality of existing features based on your latest production data on high value user journeys, expanding to adjacent use cases, lowering cost or latency.

The actual work boils down to better harnesses and model selection through methods like prompt eng, context eng, memory, post training, deterministic old school code etc.

Your failure mode taxonomy (from part 3) is a good compass for where your product struggles and needs some love.

E.g. maybe tool calling failures are your most common problem. You dig in and notice you stuff 20 tools in context, when each task really only needs 3-5. hill climbing here involves context eng to give it the right tools at the right stage and iterate until you get it to good.

Or take the cost reduction goal..
I’ve written about how I advise launching your product with the best model first. Get the quality as high as you can. Once you know users love the experience, hill climb to get similar quality with a smaller, cheaper, faster model. Same methods - harness, models.

The important thing is to have evals that tell you whether you are actually moving in the right direction.

More tomorrow.

Send this to your teammates!

Drop your questions in the comments and I will answer in future posts.

[Pascal Bertrand @PascalBtd]

@pascal_app cool name ;)

[Pascal Bertrand @PascalBtd]

@istvan_csanady We’ve found in our evals that letting the agents code doesn’t materially change outcomes for complex tasks. This comes down the point you touched: in practice, engineers intuitively run a lot of visual verifications that a model without a 3D representation of the object cannot do

[Pascal Bertrand @PascalBtd]

This will be fun!

[Seldon @seldon_tech]

We're hosting another event in San Francisco coming monday, this time a research talk with @samsja19 as guest speaker

Drop by if you're in town!

Sign up link is in the comments

[Lorenzo @lsteno]

why haven’t other engineering practices seen the same model improvements as coding? we measured models’ ability to interact with a cad environment, and the results were super interesting, with some curious upsets in model and harness rankings. i’m super excited for what’s to come in computer use

[Pascal Bertrand @PascalBtd]

@ZviadKardava @seldon_tech no this is a different one

[Pascal Bertrand @PascalBtd]

@ivanleomk There are a lot of evals missing esp. in engineering

[Pascal Bertrand @PascalBtd]

@ShubhamAg0x correct

[Pascal Bertrand @PascalBtd]

@patloeber We found the same result on CADBench

[Pascal Bertrand @PascalBtd]

@DynamicWebPaige @GoogleDeepMind @seldon_tech thanks paige!

[Pascal Bertrand @PascalBtd]

@aaxsh18 @immad made the same experience :/

[Pascal Bertrand @PascalBtd]

@aaxsh18 stay tuned for that one

[Pascal Bertrand @PascalBtd]

@seldon_tech lot's of work to do still

[Pascal Bertrand @PascalBtd]

This is pretty surprising

[Seldon @seldon_tech]

Gemini 3.7 Flash delivers performance comparable to Fable 5 at a fraction of the cost

We see that many models finish the tasks prematurely, resulting in lower scores and costs

[Seldon @seldon_tech]

How good are agents actually at CAD?

Today we release CADBench, evaluating 10 Models on 105 realistic tasks in Fusion 360

Fable 5 and Gemini 3.7 Flash are #1 with only 24.6% pass rate

[Pascal Bertrand @PascalBtd]

@jocarrasqueira @SHACK15sf @googlegemma @o_lacombe @osanseviero @DynamicWebPaige @clmt @matthewridenour was a cool event!

[Pascal Bertrand @PascalBtd]

@DhruvAhuja2003 this tweet is sponsored by the pharma industry

[Atoof @atoof_sh]

@AIatMeta @xiaolonw Teach my guy some CAD

[SkalskiP @skalskip92]

@PascalBtd thx for this link! we want to start evaluating VLMs on videos soon. articles like this are super inspiring for me.

[Pascal Bertrand @PascalBtd]

@anna_strumpel codex cowboy

[Pascal Bertrand @PascalBtd]

professional situation monitoring