Karim

Ex-quant, now building @PlatineAI. Agents should have error bars too.
@Harvard @UCLA

[Karim @pass_at_k]

@AnthropicAI That explains why my Opus 5.5 sessions this week were full of "You're absolutely right."

[Karim @pass_at_k]

@ThinkingUSD This is the reason why I don't date. Whenever I like a woman, she's already had 3 kids with @elonmusk.

[Karim @pass_at_k]

@theo Theo giving the eulogy and @TheAhmadOsman doing CPR.

[Karim @pass_at_k]

@llm_wizard @HamelHusain Nice sim, top spot seems solid! The interesting part might be the middle moving 5-6 places with only +/- 10% changes, and actually those are the models/effort levels most users and teams are working with daily. Would love to see the bigger sims if you do!

[Karim @pass_at_k]

@rachitcodes 100%. Really enjoyed this one, keep them coming!

[Karim @pass_at_k]

@rachitcodes (Forgot to mention: the 2.2 assumes each prompt differs from the baseline on a realistic 10% of the 200 cases)

[Karim @pass_at_k]

Great writeup, Rachit! One habit worth adding: keep a second set you never tune on. Hill-climb on the first as you said, and only use the second to decide if a change ships. Otherwise luck can fool you; e.g., say you try 50 prompts as good as yours on 200 cases, the best one still looks about 5 points better (chance alone moves each prompt’s score by about 2.2 points, and the luckiest of 50 lands about 2.25 times that high: 2.25 * 2.2 is almost 5).

And of course, keep refreshing that second set with new production failures so it never gets stale.

[Karim @pass_at_k]

@speedrun The most impressive things teams shipped mid-application? Asking for 30,000 founders refreshing their inbox. 🫡

[Karim @pass_at_k]

@imjustnewatai Benchmarks tell you how a @claudeai model did at launch. Your evals tell you how it's doing today. If you use it in an agent in prod, only one of those matters.

[Karim @pass_at_k]

@AnthropicAI That explains why my Opus 5.5 sessions this week were full of "You're absolutely right."

[Karim @pass_at_k]

@i2bhai8 @sermakarevich Sounds like a bot reply, but yep, that's exactly the kind of example I mean.

[Karim @pass_at_k]

Nice talk, @ajratner! Your point about goals changing mid-task is a big one for grading. Most graders check the final state against the final requirements, so an agent that did exactly the right thing at step 3 scores poorly because someone changed their mind at step 7. I’d grade two things separately: whether each action was right given what the agent knew (or should’ve asked) at the time, and once things changed, whether it noticed, fixed what it could, and flagged what it couldn’t... Lots to explore.

[Karim @pass_at_k]

@safishamsii Beautiful. Congrats, Safi!