Alex Dimakis

Professor, UC Berkeley, Founder @bespokelabsai

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Alex Dimakis @AlexGDimakis]

@RichardSocher @Recursive_SI Congratulations, incredible performance and results!

[Alex Dimakis @AlexGDimakis]

Congratulations to the team for releasing Frontier CS2.

[Qiuyang Mang @MangQiuyang]

FrontierCS 2 is our new benchmark for CS research & development.

Each task is based on a paper: we delete the contribution from the released code, and ask agents to outdo the original method, as judged by the authors' chosen metrics. Agents don't have access to the paper or web.

[Alex Dimakis @AlexGDimakis]

@BarrAlexandra hi @BarrAlexandra i am doing some research on emerging misalignment and how to create RL environments to mitigate it- happy to chat, DM open.

[Alex Dimakis @AlexGDimakis]

@asmirkn Great question. It starts very strong. Only Astra is faster and they almost match in performance from 4h to 24h.

[Alex Dimakis @AlexGDimakis]

We added Gemini 4 Argon to our AutoResearchExam Pareto curve. It is 2-3 times cheaper compared to the other frontier models for almost the same ML engineering performance. Congratulations to the GDM team.

[Alex Dimakis @AlexGDimakis]

@melvinjohnsonp Congrats on the great model and looking forward to what you’re cooking next!

[Alex Dimakis @AlexGDimakis]

@Capslock404 @madiator It means Argon is able to solve ML training and data curation problems. Specifically the problems in AutoResearchExam

[Alex Dimakis @AlexGDimakis]

https://t.co/FB1DlVOmF2 More information on AutoResearchExam

[Alex Dimakis @AlexGDimakis]

@StefanoErmon great release !

[Alex Dimakis @AlexGDimakis]

Glm 5.3 benefits come from post-training alone. Orders of magnitude cheaper than pretraining and can use proprietary data if they are transformed into RL environments.

[SemiAnalysis @SemiAnalysis_]

GLM 5.3’s biggest leap came from post-training alone.

“GLM 5.3 was built entirely by post-training the previous model. For me, that was a wake-up call.”

“Post-training is far more GPU-efficient than pre-training from scratch. Plenty of companies can’t afford to train a model from scratch, but they have proprietary data. They can post-train themselves and beat some of the closed models.”

“There’s still so much juice left to squeeze from post-training. We compared GLM 5.3’s cyber capabilities with 5.2, and it’s a night and day difference. That’s all just post-training.”

[Alex Dimakis @AlexGDimakis]

@dbreunig We're all in the same idea-basin @DimitrisPapail @ShangyinT @MangQiuyang

[Alex Dimakis @AlexGDimakis]

@sayshrey pure sorcery 🪄

[Alex Dimakis @AlexGDimakis]

@mkirchhof_ @lightetal Thanks for the pointers @mkirchhof_ we will look into them.

[Alex Dimakis @AlexGDimakis]

We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters.

The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end.

We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam.

As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections.
Surprisingly, training on these lessons improved the performance to +25% compared to the base model.

The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference.

Full write-up:
https://t.co/0lxk8PTTxW

Thanks to @NVIDIAAI for early access and collaborating on this.

[Alex Dimakis @AlexGDimakis]

@alexatallah Great work !

[Alex Dimakis @AlexGDimakis]

@RKSumedh yes it does sound like random restarts but it does much better than random restarts !

[Alex Dimakis @AlexGDimakis]

As agents perform AutoResearch they get fixated on their old ideas and do only incremental innovations. We introduce a simple context management trick that forces the agents to forget everything they have done except the best solution they have found so far.
This simple modification in context management allows the agents to get out of their idea basins and come up with more novel solutions.
Closely related to the work 'Fork and Flush' and 'Idea-basins' that @DimitrisPapail posted a few days ago.

[Shangyin Tan @ShangyinT]

Autoresearch with coding agents can now run for days on a single problem, but we've seen an increasing number of cases where they stop improving early. We address this issue with a simple idea: forgetting.

Introducing our new paper: "Scaling Discovery Requires Forgetting"

[Alex Dimakis @AlexGDimakis]

@kuchaev It post-trains pretty well: https://t.co/dKEyyNt79j

[Alex Dimakis @AlexGDimakis]

We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters.

The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end.

We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam.

As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections.
Surprisingly, training on these lessons improved the performance to +25% compared to the base model.

The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference.

Full write-up:
https://t.co/0lxk8PTTxW

Thanks to @NVIDIAAI for early access and collaborating on this.

[Alex Dimakis @AlexGDimakis]

@kuchaev Congratulations on the release !

[Alex Dimakis @AlexGDimakis]

@kuchaev @NVIDIAAI @ClementDelangue What could it possibly be ?