Xingjian Bai

Ph.D. student at MIT EECS, working with @phillip_isola and @vincesitzmann. Previous RA at @Oxford_VGG.

[Xingjian Bai @SimulatedAnneal]

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?

We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.

In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.

📝 https://t.co/augwW3iFsB
💻 Try it on your task: https://t.co/NdQNekaHoc
🧵

[Xingjian Bai @SimulatedAnneal]

@Shakhnazar98851 I agree that works on learned optimizers and learning to learn are indeed the ancestors! Agents enable zero-shot application and applicability to general tasks.

[Xingjian Bai @SimulatedAnneal]

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?

We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.

In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.

📝 https://t.co/augwW3iFsB
💻 Try it on your task: https://t.co/NdQNekaHoc
🧵

[Xingjian Bai @SimulatedAnneal]

@tokenpilled65B @tenderizzation MoE codebase is crazy lol

[Xingjian Bai @SimulatedAnneal]

Thank you Robert, very excited to see this result and your analysis!
I do agree with you that OPAL would help more in longer training runs compared with benchmarks such as modded NanoGPT or NOISE, where the training dynamics varies much from beginning to end and static optimizers are not specialized for that.

[Xingjian Bai @SimulatedAnneal]

OPAL also improves 0th-order optimization! A clean new piece of evidence found by Robert that online reactivity can power all types of optimizers.

“
There is an interesting moment at update 243: The record switches its gradient estimator (Boost → Cut), and the gradients change scale. All three agents had tuned their optimizers on Boost's gradients, and they lost ground right after the switch, up to 0.03 in training loss. OPAL noticed, rescaled the LM head's Adam statistics to the new gradients, and kept adjusting. By update ~620, every run was ahead of the record's optimizer again, and the informed runs stayed ahead to the end.
“

[Robert Lange @RobertTLange]

I was wondering if OPAL could help the optimizer for the 0-order noise val track.

So I took the (old 10M token) Noise benchmark's best validation-loss record (#14, 3.676) and wrapped its optimizer with OPAL. Every 50 updates, gpt-6.1-sol reads the loss and its own diagnostics, rewrites the update rule in Python, and tests it before it goes live.

Results (seed 0):
- record: 3.6758
- no agent (rerun): 3.6773
- agent, neutral prompt: 3.6791
- agent, prompt informed by the record: 3.6695
- same, xhigh reasoning: 3.6656 (−0.010 vs the record)

The OPAL system prompt mattered: telling the agent what its gradients are, how the record's optimizer came about, and how the end-of-run weight merge works turned no gain into a clear one.

There is an interesting moment at update 243: The record switches its gradient estimator (Boost → Cut), and the gradients change scale. All three agents had tuned their optimizers on Boost's gradients, and they lost ground right after the switch, up to 0.03 in training loss. OPAL noticed, rescaled the LM head's Adam statistics to the new gradients, and kept adjusting. By update ~620, every run was ahead of the record's optimizer again, and the informed runs stayed ahead to the end.

What the best agent changed, keeping the record's ANVIL II update for the weight matrices:
- per-layer rates for the residual mixing scalars, which it halved at the start
- matrix rate −10%, +20% on two query matrices, −15% on the value matrices
- longer momentum averaging (higher β1) for the scalars, head and gates
- the head's Adam rescale at the estimator switch

This is, of course, only anecdotal, and a minor improvement. Still, demonstrates that OPAL doesn't diverge and can mine small gains.

Intuitively, online agent-guided adaptation makes the most sense when hyperparameters/configurations haven't been tuned to death.

[Xingjian Bai @SimulatedAnneal]

@harrisonju Totally agree. Lots of things to try here! Just thinking out loud, I might start with ideas inspired by evolution or neuroscience.

[Xingjian Bai @SimulatedAnneal]

Sorry again to our infra engineers out there 😶 but changing architectures mid-training (e.g. adding and removing sub-modules) to respond to the fine-grained dynamics inside the network sounds pretty fun to me...

[Chidhambararajan R (a.k.a Chidha) @TheSeriousProg]

@jayden_teoh_ Bro optimizer perhaps can adapt live. But don't think architecture can adapt mid training

[Xingjian Bai @SimulatedAnneal]

Thanks, Marc, for the careful read! In fact, the entire strategy discovered by a reactive optimizer (the sequence of all versions of https://t.co/W3e5JF48Em it writes) can also be “crystallized” into a static recipe after training. Whether it works as well on a new run is still an open question.

[Xingjian Bai @SimulatedAnneal]

Hi, Igor 🙂 Yes, I would also describe it as “online meta-learning” of the optimizer. The “online” perspective lets agents react and adapt to specific training dynamics. I find it especially interesting that we let the agent decide what it needs to observe (https://t.co/E6KlkiJxWE) to improve.

[Xingjian Bai @SimulatedAnneal]

Hi, this is okay because it is your network, not the optimizer, that is the outcome of the training run. The network is still evaluated on held-out datasets to test its generalizability. Meanwhile, the reactive optimizer’s behavior is specialized (potentially “overfitted”). That is not a bad thing and people have been specializing optimizers to architectures a lot.

[Xingjian Bai @SimulatedAnneal]

@Aria_086 @vincesitzmann Thanks, Aria. 🙂 In our default setting, it is called 100 times during the run. It is also called to react wheneveer loss reaches NaN.

[Xingjian Bai @SimulatedAnneal]

@HenryYe19352122 Yes, we discuss the trick of running agent thinking and training steps asynchronously, which greatly reduces the overhead. Please refer to Section 4 and Figure 9 of the blog!

[Xingjian Bai @SimulatedAnneal]

@flbjcy Thank you, Jia-Hao, the pleasure is mine! Let's do some more fun work together in the future. : )

[Théo Vincent @Theo_Vincent_]

@SimulatedAnneal I love the research direction!

You might be interested in https://t.co/wipZRneIHV. It tunes the HPs, architecture, and optimizer by itself, during training.

@AlexDGoldie also worked in that direction with learned optimizers https://t.co/dMGF0jCRU2

The 2 focus on autoRL.

[Xingjian Bai @SimulatedAnneal]

Indeed, it slows down training a bit! Section 4 of the blog discusses this, and I’ve attached some additional equal-time comparison plots. You’re right that, since I make agents react infrequently in my experiments, the overhead per run is manageable: at most 2 hours and $6–$15 in tokens. In general, the relative overhead is lower for larger-scale training runs.

[Xingjian Bai @SimulatedAnneal]

@kdpisda Yes, in the way you described, we can indeed crystallize every reactive training episode into a static one. How it would perform on a different random seed of the same training is open and hasn't been studied.

[Xingjian Bai @SimulatedAnneal]

@rudraksh_tuwani Good question, we've considered this! In OPAL, there is a non-blocking mode where training and the agent run asynchronously, so the agent's reactions are slightly stale. On large training, it doesn't harm performance but improves efficiency a lot. Check Section 4 of the blog!

[Xingjian Bai @SimulatedAnneal]

@QuidVanNovi Yes, Quid, really like this perspective. The static concept of Adam could potentially go away. We talk about the ability to drive a car from home to work, but rarely about the concept of "all the decisions that need to be made while driving there".

[Xingjian Bai @SimulatedAnneal]

Current OPAL design: there is no held-out gate before a rewrite ships; any NaN loss would pause training and call the agent to react; we keep test data out of reach, tell the agent not to read it, and audit what its programs do.
Admittedly, there's no hard guardrail on the agent's reactive behavior during training. With Opus 5.5, all experiments do run stably. Perhaps that's the risk we pay for giving it max flexibility?

[Xingjian Bai @SimulatedAnneal]

@SMishra61 Honestly, I'm not sure what the best way is to keep reactive agents from making short-sighted decisions and going bad. Is trial and error part of the capability?

[Xingjian Bai @SimulatedAnneal]

Good question! Right now OPAL doesn't roll back the weights or optimizer state, so a bad rewrite's effect stays in the trajectory, and the agent can only fix or revert its program at a later decision.
Since the agent keeps one conversation for the whole run, it remembers what it changed and what it expected. In our Goldfish run it raised Muon momentum from 0.95 to 0.975, then put it back 323 updates later, after the loss rose and the weights grew.