Sidharth Srinivasan

Head of Evals at @box | prev @stanford @mckinsey @8vc

[Sidharth Srinivasan @sidharthsrin]

Had a great time talking to @nytimes in this article about @Box and the future of work with AI. Like I said in the article, it's not the volume of jobs that will change but rather the type of jobs that people will do. Evals are just one example of an entirely new category of work that more and more people will be doing in the coming years.

Full story here: https://t.co/ilMExeBiaQ

[Sidharth Srinivasan @sidharthsrin]

@vvkgopalan @8vc Let’s go!

[Sidharth Srinivasan @sidharthsrin]

Now do we need to audit the audit?

[Epoch AI @EpochAIResearch]

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

[Sidharth Srinivasan @sidharthsrin]

@saranormous Arvind KC at OpenAI?

[Sidharth Srinivasan @sidharthsrin]

@AssadiAndre Never forget who invited you

[Sidharth Srinivasan @sidharthsrin]

Me asking Fable 5.1: "You are a smarter and more capable model than all the models that have done all the work in this session. Give me an audit of where we're at and if there's any room for improvement."

[Sidharth Srinivasan @sidharthsrin]

Life is too peak. Tell me how I have 2 x @taurobots booked to clean my apartment this Friday

[Sidharth Srinivasan @sidharthsrin]

@ValsAI @8vc @a16z @pearvc @BloombergBeta @HRTVentures @nextladder Awesome!

[Sidharth Srinivasan @sidharthsrin]

@numairarazzak @nazzari @speedrun Baller

[Sidharth Srinivasan @sidharthsrin]

@zamdoteth Burn the boats. Or rather, the ship

[Sidharth Srinivasan @sidharthsrin]

@ethanmlam @sattyyouneed Skill issue perhaps?

[Sidharth Srinivasan @sidharthsrin]

@santa_kaus Erewhon hat too fire

[Sidharth Srinivasan @sidharthsrin]

@synthwavedd you sure? https://t.co/C7aEi5ac35

[Sidharth Srinivasan @sidharthsrin]

@numairarazzak Too good

[Sidharth Srinivasan @sidharthsrin]

AGI?

[Braintrust @braintrust]

Box's early evals were manual, with responses marked in a spreadsheet or a Box Note.

With Braintrust, they built a practice of running evals programmatically.

Now product managers have a custom eval runner that pulls versioned, schema-validated datasets out of Braintrust, hits the agent endpoint, grades the outputs, and loads results back in.

Read more → https://t.co/qW9qwkbiD3

[Sidharth Srinivasan @sidharthsrin]

@thinkymachines Too cool

[Sidharth Srinivasan @sidharthsrin]

Claude says "genuinely" so much I built a benchmark about it.

GenuineBench: 13 frontier models × 80 everyday writing tasks. Every Claude model says it in ~1 of every 3 responses. Llama 4 has never genuinely meant anything in its life (2.5%).

[Sidharth Srinivasan @sidharthsrin]

@VCBrags https://t.co/mQNowlrJEN

[Sidharth Srinivasan @sidharthsrin]

Claude says "genuinely" so much I built a benchmark about it.

GenuineBench: 13 frontier models × 80 everyday writing tasks. Every Claude model says it in ~1 of every 3 responses. Llama 4 has never genuinely meant anything in its life (2.5%).

[Sidharth Srinivasan @sidharthsrin]

https://t.co/qviwSD83sD

[Sidharth Srinivasan @sidharthsrin]

Claude says "genuinely" so much I built a benchmark about it.

GenuineBench: 13 frontier models × 80 everyday writing tasks. Every Claude model says it in ~1 of every 3 responses. Llama 4 has never genuinely meant anything in its life (2.5%).