Lin Gui

Phd @Uchicago

[Lin Gui @ybnbxb]

@androneil54 @HeMuyu0327 That makes sense. Thanks!

[Lin Gui @ybnbxb]

@androneil54 @HeMuyu0327 Not exactly right when pi_theta=pi_old?

[Zhuokai Zhao @zhuokaiz]

On-policy RL has driven the biggest leaps in training coding agents. Extending it to machine learning engineering agents should be a natural next step.

But it almost never works.

What I mean is, the recipe is right there — standard trajectory-wise GRPO, the same that worked for SWE.

However, the problem is that one rollout step on an MLE task may take hours because the agent has to actually train a model on a real dataset at every step (preprocessing, fitting, inference, scoring). So even with the N rollouts in a group running in parallel, a single GRPO run may still take days.

Every MLE agent paper I've read has retreated to SFT or offline proxy rewards for exactly this reason, giving up the exploration benefits of on-policy learning.

That's why I'm excited about our new paper, SandMLE, which fixes this with a move that sounds almost too reckless to work.

The instinct when on-policy RL is too slow is to engineer around it — async rollouts so the trainer doesn't sit idle waiting for slow environments, off-policy or step-wise proxies to avoid running full trajectories at all.

But when we profiled where the time was going, the bottleneck had nothing to do with the algorithm.

Unlike SWE where execution latency comes from compilation and test logic, MLE latency is overwhelmingly driven by the size of the dataset the ML pipeline has to chew through.

Therefore, rather than downsampling existing data (which corrupts evaluation), we built a multi-agent pipeline that procedurally generates diverse synthetic MLE environments from a small seed set.

Specifically, we extract the structural DNA of seed tasks (modality, label cardinality, distribution shape), mutate them into new domains (e.g., repurposing animal classification into road damage detection), inject realistic noise, embed deterministic hidden rules connecting features to labels, and construct full evaluation sandboxes with progressive milestone thresholds.

Each task is constrained to only 50–200 training samples.

The execution speedup is dramatic — average per-step latency drops over 13×, which makes trajectory-wise GRPO go from infeasible to routine.

We also designed a dense, milestone-based reward to address the sparse credit assignment problem in long-horizon MLE. The ablation shows this matters — under a sparse reward, the 30B model's medal rate drops from 27.3% to 13.6% and valid submission collapses from 100% to 86.4%.

Results across Qwen3-8B, 14B, and 30B-A3B on MLE-bench are consistently strong — 66.9% better performance in medal rate over SFT baselines.

It is worth noting that the SFT baselines are not weak— we trained them on high-quality Claude-4.5-Sonnet trajectories.

But SandMLE still delivers much larger gains, suggesting that direct environment interaction does teach capabilities that imitation alone does not (as expected).

The most convincing evidence to me that the model's intrinsic performance gets improved is the framework-agnostic generalization.

We trained exclusively with ReAct but the gains transfer to AIDE, AIRA, and MLE-Agent scaffolds at evaluation time — up to 32.4% better performance in HumanRank on MLE-Dojo.

The SFT models, by contrast, are brittle when moved to unfamiliar scaffolds. The 30B SFT model collapses to 17.7% valid submission rate on MLE-Dojo with MLE-Agent, while the 30B SandMLE model achieves 83.9%.

SandMLE is teaching genuine engineering reasoning, not scaffold-specific patterns.

What I find most interesting beyond the specific result is that none of the hard parts of RL changed here.

The algorithm is the same. The reward is conventional. We just shrunk the environment until on-policy learning became affordable.

The field has largely treated environment design and RL algorithm design as separate concerns.

SandMLE is a concrete case that the environment is itself the lever.

When training is too expensive, the instinct is to build cleverer algorithms to tolerate it.

However, often the better move is to reshape the environment so the simple algorithm just works.

Paper: https://t.co/x0jAvCClyh

[Yinjie Wang @YinjieW2024]

OpenClaw-RL Technical Report! Make your🦞@openclaw stronger by just using it. We propose a method that combines the advantages of GRPO and OPD, and evalution results. The repo is already 1.7k stars now, feel free to contribute! Come in and have fun~
@MengdiWang10 @LingYang_PU

[Lin Gui @ybnbxb]

@cwolferesearch Also check out our paper: https://t.co/wUtyLHpm43!

[Zihao Wang @wzihao12]

On-policy distillation with reverse KL as reward works great—IF you have access to teacher logits.
But what if you don't? What if you want to distill from multiple teachers?
Our solution: distill teacher guidance into rubrics, then do on-policy RL.
Check out our work: https://t.co/5kVYSSUGOB

[Chenghao Yang @chrome1996]

Where is exploration most impactful in LLM reasoning? The initial tokens! They shape a sequence's entire semantic direction, making early exploration crucial.

Our new work, Exploratory Annealed Decoding (EAD), is built on this insight. By starting with high temperature and cooling down, we encourage meaningful diversity early while preserving quality later.

The result is a simple, plug-and-play method that improves sample efficiency and provides consistent gains in RLVR across various models and RL algorithms. EAD also boosts performance for inference-time scaling.

📜 Paper: https://t.co/tXw4EeJlxM
🗳️ Upvote on Daily Papers: https://t.co/Vho9GVH1LB
🌐 Website: https://t.co/jJM7xuDRIw
Code: https://t.co/coH5kETDOM

Great thanks to my amazing collaborators! @ybnbxb , @chenxiao_yang_ @zhuokaiz @victorveitch
Also thanks for the great support from @DSI_UChicago @TTIC_Connect @Meta !

[Bing Liu @vbingliu]

New @Scale_AI paper!

The culprit behind reward hacking? We trace it to misspecification in high-reward tail.

Our fix: rubric-based rewards to tell “excellent” responses apart from “great.”

The result: Less hacking, stronger post-training!  https://t.co/D6aJkZ8zZE

[David Reber @davidpreber]

🧵 RATE: Score Reward Models with Imperfect Rewrites of Rewrites
1/ How do you measure whether a reward model incentivizes helpfulness without accidentally measuring length, complexity, etc?
Rewrites of rewrites give good counterfactuals, without needing to list all confounders!

[Yibo Jiang @yibophd]

Are LLMs just doing next token predictions? It is believed that if an LLM can accurately predict the next tokens in a Wikipedia entry, it essentially "learns" the information.

But do pre-trained LLMs actually need to understand context sentences to solve this task? The answer is no!

[Victor Veitch 🔸 @victorveitch]

LLM best-of-n sampling works great in practice---but why?

Turns out: it's the best possible policy for maximizing win rate over the base model!

Then: we use this to get a truly sweet alignment scheme: easy tweaks, huge gains

w @ybnbxb @ggarbacea

https://t.co/GHaP911Lil

[Yibo Jiang @yibophd]

How is the semantic structure of text encoded in the algebraic structure of embeddings? In our #NeurIPS2023 paper, we show that ``semantic independence'' in language is represented as partial orthogonality of embeddings! https://t.co/TSQeuc1mXz w @itsrainingdata and @victorveitch

[Zihao Wang @wzihao12]

Prompt engineering is a dark art. To understand limitations, we formalize "what the user intended" through latent concepts. Then we show direct model controls by algebraic operations on suitably chosen representations!https://t.co/kK4I38Jf76 w @ybnbxb @jeffNegrea @victorveitch

[Lin Gui @ybnbxb]

Will present this work at Causal ML for Real-World Impact workshop today at NeurIPS. Welcome to stop by in room 295-296:)

[Lin Gui @ybnbxb]

How can we use modern language models to understand the causal effects of text? E.g., does being polite in email cause faster responses? This one weird trick lets you do robust causal effect estimation with text data! https://t.co/teSo18Uvr8 w @victorveitch

[Lin Gui @ybnbxb]

Our method turns out to be robust in the sense that low bias and valid uncertainty quantification can be expected with misestimation of conditional outcomes.

[Lin Gui @ybnbxb]

How can we use modern language models to understand the causal effects of text? E.g., does being polite in email cause faster responses? This one weird trick lets you do robust causal effect estimation with text data! https://t.co/teSo18Uvr8 w @victorveitch

[Lin Gui @ybnbxb]

Solution2: we learn some data representation of text satisfying both unconfoundedness and overlap by supervised representation learning and use it as the confounding part. We then apply non-parametric models for propensities and the double-ML method to obtain the final estimator.

[Lin Gui @ybnbxb]

Problem 2: overlap violation means we can't use double-ML methods out of the box. Can we still get robustness to slow rates from the language model?

[Lin Gui @ybnbxb]

Problem: what is even the causal estimand when we don't have overlap?
Solution: we pick a well-defined controlled direct effect that captures the meaning of "causal effect of an attribute of text”. It is estimable under some mild assumptions. (See Section 2 and 3 of the paper)

[Lin Gui @ybnbxb]

Q: Why does overlap violation happen?
A: An attribute of the text is determined by the text. So if we directly use the text as the confounding part, P(Treatment=1|text)=0/1.