Fabian Schaipp

working on optimization for machine learning. currently postdoc @inria_paris.

[Fabian Schaipp @FSchaipp]

Learning rate schedules seem mysterious?
Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization.

Short thread on our latest paper 🚇

https://t.co/DGHoG1FS3f

[Aaron Defazio @aaron_defazio]

The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting!
1/2

[Fabian Schaipp @FSchaipp]

@ruuustem_10 @orvieto_antonio @n_ajroldi yeah, agree! i guess you can get sth similar for the smooth case too, but probably need to adapt the alpha< 2/L step

[Fabian Schaipp @FSchaipp]

"It's easier to tune the LR for method A than for B."

We tried to formalize this for model-based stochastic optimization methods.

We find a key quantity, called stability index, that describes how stable a (weakly) convex bound is as a function of LR.

📚https://t.co/JIrG0gXqXL

[Fabian Schaipp @FSchaipp]

The theory is with SGD as base update step. But all these adaptive step sizes can be used to obtain practical methods. For example, Polyak step sizes in combination with

- Muon (@CrichaelMawshaw): https://t.co/ztFcHNWzmL
- ScheduleFree (@aaron_defazio ): https://t.co/HBsVJ20RjJ

[Fabian Schaipp @FSchaipp]

Joint work with @gowerrobert and @TaylorAdrien , and just accepted at #ICML2026.

[Fabian Schaipp @FSchaipp]

Polyak step size is back!

[Aaron Defazio @aaron_defazio]

🚨 New Paper 🚨
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

A few modifications to Schedule-Free Learning make it completely LR tuning free, and allow it to greatly outperform schedules for long duration training!
https://t.co/LzjIIsOlG8

[Fabian Schaipp @FSchaipp]

@maxzimmerberlin @spokutta Glückwunsch, Dr. Zimmer!

[Fabian Schaipp @FSchaipp]

NeurIPS 2025 Proceedings are finally online!

[Fabian Schaipp @FSchaipp]

@konstmish when I tested CWD, it looked much better than the baseline for a long time, then lost all its advantage during cooldown. 😢
Was for a relatively short run though (D~20N).

[Fabian Schaipp @FSchaipp]

Nice result! (from https://t.co/O8sr1pvZhe)

no anytime-schedule can obtain the optimal rate for (S)GD.

to my knowledge, WSD is the closest candidate we know of, as it removes the log-factor in the rate for any cooldown length proportional to T.

[Fabian Schaipp @FSchaipp]

Going to Zurich for a couple of days. I will give a talk on recent optimization stuff @zurichnlp. Always happy to chat 🍫

[Fabian Schaipp @FSchaipp]

@ruuustem_10 @YouJiacheng @CevherLIONS not sure i follow. the steplaw is not restricted to a fixed TPP.

[Fabian Schaipp @FSchaipp]

@CevherLIONS @YouJiacheng thanks for clarifying! so the batch size scaling is also only applicable to Scion?

[Fabian Schaipp @FSchaipp]

@JFPuget yes!

https://t.co/HoNlIwhv1Q

[Fabian Schaipp @FSchaipp]

@rishabh16_ Muon for material foundation model: https://t.co/vNXpOOtkEU

Not Bio, but closely related: https://t.co/q87mHqymyj

[Fabian Schaipp @FSchaipp]

After LLMs and diffusion, Muon also shines on tabular foundation models!

Also nice to see they used cautious weight decay 🥌

[David Holzmüller @DHolzmueller]

Super excited that TabICLv2 is out 🎉

🚀Beats RealTabPFN-2.5 with no tuning and purely synthetic pre-training data.

👉Introduces QASSMax for long-context generalization, early target embedding, repeated feature grouping, Muon, etc., and a much diversified synthetic data prior.

[Fabian Schaipp @FSchaipp]

@ADarmouni that's not how research works

[Fabian Schaipp @FSchaipp]

not to offend anyone, but how tf do these papers get through review when not even the LR of the baseline is properly tuned?

[Zain @ZainHasan6]

Learning rate matters more than your LoRA variant.

In this study they sweep LR hard across LoRA variants (DoRA, Init[AB], PiSSA, MiLoRA) and find:

> If you tune LR properly, they all converge to approx the same peak perf.

> Rank still matters and can flip which variant looks best depending on dataset.

> Optimal learning rate is a function of how steeply curved the loss is:

>> more curved → smaller steps (lower LR)
>> less curve → larger steps (higher LR)