Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: https://t.co/ZdeshNHhcy
Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities.
In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code).
Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.
I've been thinking about where the data for the next generation of open models will come from.
The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. https://t.co/tSajZlkEh9
If you're keeping tally of our blogs @ravwojdyla enabled:
- ~All the infra that enabled our data scale
- A kernel that let us ~2x our parameter count while maintaining higher MFU
🐐
Marin’s 535B-parameter hero run was originally planned as a 360B model. In https://t.co/zkCMPu3bZG we explain how expert parallelism made room for a roughly 50% larger model, with approximately 23B active parameters per token - and the tradeoffs involved.
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
It’s been a few weeks since Marin kicked off our 535B/A23B MoE hero run. So far it’s gone almost boringly well.
It’s also been just over a year since Marin joined Open Athena. I wrote up some thoughts about how far we’ve come, and how we got there. https://t.co/K9JW1ZrfHb
I wrote about why more of that process knowledge should become public, and why Marin is organized around open development of frontier AI.
https://t.co/Qcu5R7kger
Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions. https://t.co/5dPB9uBiSp
To train better open models, we need predictable scaling.
Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error.
Getting there took some work 🧵
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
So about a month ago, Percy posted a version of this plot of our Marin 32B pretraining run. We got a lot of feedback, both public and private, that the spikes were bad. (This is a thread about how we fixed the spikes. Bear with me. )
What would truly open-source AI look like? Not just open weights, open code/data, but *open development*, where the entire research and development process is public *and* anyone can contribute. We built Marin, an open lab, to fulfill this vision: