Siddartha Devic ✈️ COLM 2026

Member of Technical Staff @bespokelabsai. PhD @CSatUSC in trustworthy AI/ML, previously intern @apple @amazon, ug @UT_Dallas

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

Like (nearly) everyone else who worked on theory at some point in their academic career, I have some thoughts on the recent developments in AI and how it may impact the way that we perform and evaluate theoretical research.

[Mahesh Sathiamoorthy @madiator]

Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants.

On the Decision Index public scores, nimble-v4-max and nimble-v4 score +10 and +6 against Jev 1.13, making nimble-v4-max the #1 model.

Beyond benchmarks, we have done a lot of internal evals, and see meaningful improvements: 43% fewer mistakes than Jev, 50pp improvement on model routing decisions, being great at coding use cases, fact checking etc.

The models are available via our API service.

Go build nimbly: https://t.co/nFnE4dn7OQ

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

@BeepulBharti @bespokelabsai yes! I'll be at the @bespokelabsai booth from Noon-6pm ish tomorrow, feel free to swing by.

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

Hi friends, i'll be at COLM this week, please let me know if you are around and want to meetup! Will also be at our booth @bespokelabsai, happy to chat about opportunities we have on RL envs, post-training, and understanding agent behaviors.

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

check out our new autoresearch benchmark!

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

This is my first substack post after finishing my PhD. It was definitely useful to clarify my own thoughts about how things are moving! Here's the full article: https://t.co/Lf0PbCsneZ

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

Like (nearly) everyone else who worked on theory at some point in their academic career, I have some thoughts on the recent developments in AI and how it may impact the way that we perform and evaluate theoretical research.

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

Lastly, as a reason for optimism, I discuss a bit about how "superintelligent" coding agents have actually _increased_ collaboration between my coworkers and I, and how this may also happen for cs theory research.

[Bespoke Labs @bespokelabsai]

https://t.co/DnEKgOZC3h

[Daniel Litt @littmath]

STEM academia serves two closely intertwined purposes: the production of high quality science and the production of human capital. These two purposes feed into each other. The obvious direction is that we develop human capital by paying people to produce science.

What is perhaps less obvious is that the very fact that human labor is used to produce science has historically been an important input to its quality. The goal of science is not simply to produce papers, but rather to produce good work--that a person is willing to spend months working on a paper is a (weak) witness to the fact that it has some minimum quality. If someone has a record of producing high quality work, that they wrote a paper is a stronger witness, since it was worth the opportunity cost to write it. If many people engage with it substantially, that is even stronger evidence. This is not to say that there isn't lots of low-quality work--there is, in fact a huge amount--but we have strong sorting mechanisms, admittedly using imperfect proxies (all depending on costly human labor!), to find high-quality stuff. Arguably the paper itself is not the primary product here; in many cases the primary product is actually the expertise developed over the course of producing it, which can then be applied to other questions.

If you believe, as I do, that producing high quality science should be one of our fundamental goals, I think you’re obligated to embrace new tools that help one do so. Refusing to is a declaration that these outputs are not important. But I worry that we are not on track to automate the production of good work; rather, we are on track to automate the production of papers. We need new mechanisms to ensure that we are also producing good work, and to ensure that we are developing the human capital to engage with it.

[Bespoke Labs @bespokelabsai]

We’re thrilled to announce a $40M investment that will fuel our mission to make AI agents reliable.

For the past two years, we've been heads-down doing world-class data curation research and shipping best-in-class reinforcement learning environments for training and optimizing AI agents. This funding lets us go a lot deeper on both.

Thank you to our investors @Wing_VC, @MayfieldFund, @8vc, @thehousefund and our angels such as Jeff Dean, Dheeraj Pandey, Tristan Handy, and several others from Anthropic, OpenAI, Meta. And thanks to the frontier labs and enterprises we work with every day, for sharing our vision for a future where agents can run autonomously for weeks and months at a time.

(more below)

[Richard Zhuang @RichardZ412]

How can we train small agentic models that are highly capable of terminal use and coding?

Announcing OpenThoughts-Agent + OpenThinkerAgent-32B, the strongest Qwen-3 based open-data agentic model: 44.8% avg across 7 agentic benchmarks! (1/n)

[Siddartha Devic ✈️ COLM 2026 @sid_devic]

@suchenzang In at least one (popular) stylized model, identification is actually more difficult than generation: https://t.co/wqb0nBx7xb. @AnayMehrotra is one expert in this area that I know of!

[Arthur Liang @arthliang]

I’m really excited to be @iclr_conf this week presenting our work on substructure-aware protein modeling! DM me if you want to chat about protein models, baking biological priors into ML architectures, and anything bioML for experimentalists!

We built Magneton, a model-agnostic framework that distills decades of curated substructure knowledge into any pretrained encoder. We find that substructure signal is complementary to both sequence and structure as well as helps models generalize to unseen substructure types. Come say hi at our poster on Saturday from 10:30AM-1:00PM @ Pavilion 3 P3-#1006!

This work is near and dear to my heart because it’s my first (co)first-author main conference paper from my undergrad. A heartfelt shoutout to my mentors Robert Calef, @manoliskellis, @marinkazitnik for all their support!

Website: https://t.co/bvC9WM5lWZ
Paper: https://t.co/3hYEbmEc0V

[Deqing Fu @DeqingFu]

New paper: Convergent Evolution: How Different Language Models Learn Similar Number Representations.

Language models, classical word embeddings, and even raw token frequencies all develop the same Fourier features for numbers. But only some develop the underlying structure. 🧵