Vlad

Pretraining LLMs @reflection_ai // opinions are my own

[Vlad @TheVladSavinov]

Inspired by @BlancheMinerva 's Directory of Generative AI!
wanted to have something similar for recent models so it's easier to find a detail without opening HF or rereading tech reports.

PRs with new models or corrections are welcome: https://t.co/waALuKIG1Z

[Vlad @TheVladSavinov]

Built a tool/table to compare published pretraining recipes and open-weight model architectures: https://t.co/q2re9Gu6w0. All the details are in one place: width/depth ratio, KV cache per token, expert sizes, licenses, etc. (parsed from HF configs and tech reports)

[Vlad @TheVladSavinov]

@brendanm0407 🤝

[Vlad @TheVladSavinov]

building a foundation model from 0 to 1 is a rare experience these days, and I feel very lucky to be a part of it :) It’s been a pleasure pretraining this model alongside such a great team over the past 8 months, happy to see it out in the world!

[Reflection @reflection_ai]

Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.

- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch

Full weights release this month.

Learn more about Beam: https://t.co/c3Qx2cpM8G

[Vlad @TheVladSavinov]

@rvarm1 it’s because it’s yours

[Vlad @TheVladSavinov]

@iamgrigorev thanks!!

[Vlad @TheVladSavinov]

@ariG23498 how about two blogposts with different perspectives? one would be building a high level intuition, the other one deriving math

[Vlad @TheVladSavinov]

@difficultyang yep, exactly, I think this is ~something that they do

[Vlad @TheVladSavinov]

great stuff!!

one nit on the expert diagram: I think swiglu doesn't consume e4m3 directly since DeepGEMM outputs bf16, and they probably fuse quantization into the swiglu kernel

deepseek open sourced their perfetto trace for v3 in early 2025 https://t.co/FjsC5Wb2dJ. Lots of cool stuff there, for example, the swiglu time matches topK*seqlen*intermediate*(2 + 2 + 1 + 1 + 1)/2.8e12 = 8*4096*2048*7/2.8e12 = 0.16ms. IMO most likely they read two bf16 tensors and output three fp8 tensors: two of them are probably quantized inputs, and the third one is quantized `silu(a)*b` to feed into the next DeepGEMM call.

(2.8e12 above is roughly 85% from 3.35 TB/s from the spec on hoppers)

[difficultyang @difficultyang]

Attention: post two of the DSv3 series is out: https://t.co/2dkzFeBg55

[Vlad @TheVladSavinov]

3/ If something feels off - please share your feedback!

it’s also very cool to work on something like this with agents - they make it much easier to turn ideas into visuals

[Vlad @TheVladSavinov]

Interactive website to help build a mental model of distributed LLM training - drag and drop compute kernels and collectives to create DDP/TP/FSDP/EP/CP traces based on real torchtitan profiles!

There are many great resources on scaling LLMs out there, but I felt like there was still a gap when it comes to visualising different parallelisms and building intuition around overlaps and execution order. This is my attempt to make those ideas easier to see and reason about.

[Vlad @TheVladSavinov]

2/ https://t.co/EOAdPa0alY

I think of it as an interactive addition to great resources such as the Ultra-Scale Playbook by @Nouamanetazi or How to Scale Your Model by @jacobaustin132, highly recommend reading them beforehand!

[Vlad @TheVladSavinov]

@eliebakouch "num_layers": 38 👀

[Vlad @TheVladSavinov]

@BlancheMinerva according to https://t.co/2cATWx0I6M, if you fix D and not D/P, you get a higher loss penalty for a large model compared to the small one

[Vlad @TheVladSavinov]

@fzmushko would be interesting to see model-health metrics as well - curious about the impact on expert utilization/grad norms

[Vlad @TheVladSavinov]

nice work showing that delayed muon works pretty well with async PP

also using qwen-next with GDNs for ablations 👀

[Philip Zmushko @fzmushko]

We’re excited to share our #ICML2026 paper on optimizer robustness in Async Pipeline Parallel LLM pretraining

Main message: with the right PP schedule, optimizer choice, and Error Feedback correction, Async PP can match sync training quality even at 10B MoE scale

[1/10]