Philip Zmushko

PhD at @ISTAustria @EllisForEurope

[Max Ryabinin @m_ryabinin]

New paper for those working on pipeline parallelism / modern optimizers. Authors train 10B MoE in the async setting with no gap vs sync, the recipe is Muon+error feedback+the right pipeline schedule

TL;DR:
- Regular async pipelining is not widely adopted, despite solving the bubble problem. The reason is that stale gradients harm convergence, which makes larger async runs risky
- You can use schedules that bound staleness: PipeDream-2BW (use old/new copies of weights, accumulate grads and apply periodic updates/swaps) and WPipe (finer-grained splitting that halves the number of buffers/stale gradients). When these methods were proposed, the gap from 1-step delay was still thought to be too big in practice
- But, once you swap Adam with newer methods (e.g., Muon or SOAP), the gap gets much smaller. Momentum also plays a big role: regardless of the optimizer, increasing its coeff mitigatesstaleness
- Authors also apply the idea of error feedback to async updates, basically just correcting the previous update with the new one. The cost is keeping one more buffer for old updates, but it vanishes with FSDP. Surprisingly, EF+bounded staleness closes most of the async gap, for Muon and even AdamW!
- Taking all of this together, authors run larger experiments with 2B and 10B models. My main takeaways are: async+EF has the same trends wrt training budget, at 10B MoE/200B tokens there is no difference in loss to synchronous methods

Would be curious to see how this scales to even bigger runs + how much we gain in throughput, but overall a very solid study!

[Philip Zmushko @fzmushko]

Huge thanks to my amazing co-authors for their hard work!
@egor__petrov @head__water Mikhail Khrushchev @sam_hrvth

Presenting at ICML main conference + Protocol Learning Workshop @Pluralis

📄 arXiv: https://t.co/qaDQNqzEny
🤗 HF Daily Papers: https://t.co/C9qpMhqRea

[10/10]

[Philip Zmushko @fzmushko]

We’re excited to share our #ICML2026 paper on optimizer robustness in Async Pipeline Parallel LLM pretraining

Main message: with the right PP schedule, optimizer choice, and Error Feedback correction, Async PP can match sync training quality even at 10B MoE scale

[1/10]

[Philip Zmushko @fzmushko]

Finally, we validate the recipe on a 10B MoE trained for 200B tokens with Muon. Async PP + EF exactly matches the synchronous baseline in final loss, thus marking the first successful demonstration of Async PP training at this scale. 🚀

[9/10]