Shreyas Pimpalgaonkar

founding eng @bespokelabsai

prev @citadel @nyu @goldmansachs @iitbombay

[Shreyas Pimpalgaonkar @sayshrey]

Excited to share that we've raised $40M in funding!

For the past one and half years I've been working on some of the hardest problems to make AI agents reliable: data curation, RL environments and post-training.

Thanks @madiator and @AlexGDimakis for this incredible opportunity, it has been a great journey working with you and the team, can't wait for what's next!

We're hiring, please reach out if you're interested

[Bespoke Labs @bespokelabsai]

We’re thrilled to announce a $40M investment that will fuel our mission to make AI agents reliable.

For the past two years, we've been heads-down doing world-class data curation research and shipping best-in-class reinforcement learning environments for training and optimizing AI agents. This funding lets us go a lot deeper on both.

Thank you to our investors @Wing_VC, @MayfieldFund, @8vc, @thehousefund and our angels such as Jeff Dean, Dheeraj Pandey, Tristan Handy, and several others from Anthropic, OpenAI, Meta. And thanks to the frontier labs and enterprises we work with every day, for sharing our vision for a future where agents can run autonomously for weeks and months at a time.

(more below)

[Shreyas Pimpalgaonkar @sayshrey]

@ValsAI Dijkstra is not the fastest shortest path algorithm, could you please compare your results with this paper: Breaking the Sorting Barrier for Directed Single-Source Shortest Paths (https://t.co/SeB7zOw2Fo)

[Shreyas Pimpalgaonkar @sayshrey]

@jkeatn wow, is code the latent space for all forms of expression?

[Shreyas Pimpalgaonkar @sayshrey]

@HeathDingwell @OfficialLoganK GDP Val, AA-Briefcase v1.1

[Shreyas Pimpalgaonkar @sayshrey]

@OfficialLoganK @abenz95 Do you have any go to resources for finding new benchmarks?

[Logan Kilpatrick @OfficialLoganK]

if you are building a product using AI, you should be spending >25% of your time making benchmarks and trying to get the model labs to care about said benchmarks

easiest path to accelerate your progress as a company

[Qiuyang Mang @MangQiuyang]

🧵 With unlimited compute, how fast can agents surpass humans? We introduce Elo-per-token analysis to profile agent performance curves across multiple open-ended tasks.

• Agents initially scale faster than repeated sampling, but over long horizons converge toward their theoretical log-linear scaling curve. Humans, in contrast, improve superlinearly.

• These curves also tell us how to spend test-time compute: the scaling inflection point gives a simple rule for splitting a fixed budget across agent sessions. Split a long run in a principled way, and you can get significant gains over a single run.

• Fitting human-time and agent-token curves also gives us a fun way to translate AI compute into human time. Taking OpenAI’s ~130B-token Navier–Stokes run as input and extrapolating across the two curves gives an equivalent of ~41 years of work by a mathematician at 8 hours/day 😮.

[Alex Shaw @alexgshaw]

Fun fact, Harbor OSS processes more tokens per week than OpenRouter (80T vs. 70T)

[Shreyas Pimpalgaonkar @sayshrey]

As agents tackle increasingly long-horizon, open-ended problems, first-attempt performance tells only part of the story. We also need to measure how effectively they learn and improve across subsequent attempts. We find that Astra is the hare, and Fable is the tortoise! 🐢

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Shreyas Pimpalgaonkar @sayshrey]

@RichardZ412 @AnthropicAI Congratulations @RichardZ412 !

[Shreyas Pimpalgaonkar @sayshrey]

@StevenDillmann Congratulations!

[Shreyas Pimpalgaonkar @sayshrey]

@adcock_brett When participating in the robot games?

[Shreyas Pimpalgaonkar @sayshrey]

@SeanZCai yeah! any good resources on training on traces?

[Shreyas Pimpalgaonkar @sayshrey]

it's actually so surprising that AI writes so poorly when literally all the data on the internet that went into training it (at least initially) was written by humans

[Shreyas Pimpalgaonkar @sayshrey]

@leopoldasch congratulations!

[Shreyas Pimpalgaonkar @sayshrey]

Working on data eng bench was a great experience!

Domain specific evals are the key to unlocking the next frontier, reach out to us if you're interested!

[sridhar @RamaswmySridhar]

We created data-eng-bench with @bespokelabsai and we’re open-sourcing it.

There are plenty of model performance benchmarks for code generation. But for data engineering, the harness matters just as much as the model.

We built a benchmark that asks agents to build and fix real pipelines, then grades them on whether the output actually works.

The results: Using the same Opus 5 model, @Snowflake CoCo achieves 73.8% Pass@1 at 3.9× lower cost than Claude Code. With Sonnet 5, CoCo delivers the same quality at 2.3x lower cost.

Now anyone can use data-eng-bench to evaluate model + harness combinations on real data engineering workflows. Run your own tests and share what you find.

https://t.co/wlFwOZ3Ne1