Jordan Sassoon

Pre-training Efficiency @Aleph__Alpha

[Jordan Sassoon @resiliem]

What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.

I explain it in detail here: https://t.co/MtjwZIQ7AD
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.

I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.

[Aleph Alpha @Aleph__Alpha]

Tech report is available: https://t.co/F2NhzMbTDR

[Pit Neitemeier @PitNeitemeier]

One of my favourite things to see here is how for DeepSeek at short seqlens we read the attention state more than once (dashed read line above solid total) due to cross-layer reuse. Past 16K, sparse attention takes over, so we read it less than once and the lines cross.

[Jordan Sassoon @resiliem]

@alessio__serra Would be nothing without you guys 🫶🫶

[Jordan Sassoon @resiliem]

What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.

I explain it in detail here: https://t.co/MtjwZIQ7AD
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.

I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.

[Aleph Alpha @Aleph__Alpha]

Tech report is available: https://t.co/F2NhzMbTDR

[Jordan Sassoon @resiliem]

@m_newhaus @Aleph__Alpha Happy you liked it 🙌 a lot more optimisations had to be pulled off to get to 35% MFU in the first place, but our approach never changed, even through model iterations!

[Jordan Sassoon @resiliem]

@PitNeitemeier Yep, gotta watch out for your kernels. The rabbit hole is always deeper than one might think.

[Jordan Sassoon @resiliem]

@PitNeitemeier @alessio__serra Hahahahah would have loved to collab on some memes, next time ;)

[Pit Neitemeier @PitNeitemeier]

How many of your active parameters go to attention, and how many to the experts? In our new blog we derive how modern architectures allocate parameters and FLOPs, and what that means for attention state and long-context FLOPs.

Also: cute kolibris!

[Alessio Serra @alessio__serra]

here we share some of the tools we used to shape Kolibri’s architecture.

they help us understand the trade-offs before committing to choices that become expensive at scale.

thought the community might enjoy playing with them too :)

[Jordan Sassoon @resiliem]

@radke149 That’s true, Olmo’s dataset was very important also for us. But I doubt we’ll open source the corpus :/ if this changes, I’m sure you’ll see it publicly announced!

[Quantinine @quantinine]

Congrats on shipping Kolibri-1 @Aleph__Alpha 🐦

I ran ~29k graded agent episodes in German & English, comparing Kolibri-1 with Gemma 4 and Qwen3.6.

Findings your team may find useful:
https://t.co/nvMRTXE9bK

Happy to share more. @IlhanScheer @tfburns @letiepi @alessio__serra @resiliem @tradeqvest

[Jordan Sassoon @resiliem]

@quantinine @Aleph__Alpha @IlhanScheer @tfburns @letiepi @alessio__serra This is really, really interesting. Thanks for running these evals!

[Ilhan Scheer @IlhanScheer]

Kolibri, day 4.

#1 trending text generation model on Hugging Face.
30 community builds.
First GGUF went up 11 hours after release.
Runs on Apple Silicon, llama.cpp, Blackwell, a single 4090.

The most downloaded version isn't ours anymore.

Built by @Aleph__Alpha. Now built on by everyone.
That's how open weights should work.

[Jordan Sassoon @resiliem]

@naatyu0 Thank you 🙏 we use perfetto.ui for our traces, but in the blog I just made some simplified custom visuals

[Konark Modi @konarkmodi]

We gave @Aleph__Alpha ‘s Kolibri-1 13 choices and put it in Doom. What could go wrong? 🎮

Up to 72 action combinations. Four decision groups, one batch.

39ms median. 60ms p95 in our test.

Apparently enough time to make questionable choices 🫣

Come judge its aim @IlhanScheer @PitNeitemeier @alessio__serra

👇 https://t.co/HV3MWejJgg

[Jordan Sassoon @resiliem]

@abheer @Aleph__Alpha happy to hear you enjoyed the blog post 🙌

[Abheer Singh @abheer]

there is wisdom in being resourceful with your compute. fantastic blog from @resiliem @Aleph__Alpha, hierarchical scaling masterclass! learned a lot

[Jordan Sassoon @resiliem]

@Aflah02101 🤗🤗

[Jordan Sassoon @resiliem]

@radke149 As for data corpus, I don’t think there is a plan to open source that. Sorry to disappoint 😬
I believe the report shows what we trained on in detail, so you could reproduce it if you have the compute, or draw inspiration and scale to whatever your compute can handle

[Jordan Sassoon @resiliem]

@radke149 Custom trace analyzer was just a drop in replacement for perfetto.ui so readers could interact with the trace instead of having a screenshot. Claude was very helpful there 😌 but we use perfetto.ui to analyse traces in prod