Matt Wittmann

🐂 shaving extraordinaire

[Larry Dial @classiclarryd]

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: https://t.co/ZdeshNHhcy

Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities.

In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code).

Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

[David Hall @dlwh]

It’s done! Marin’s Mochi 535B-A23B finished its main pretraining phase this morning. 18T tokens! Long context this weekend, SFT over the next week or so. Hoping to release checkpoints next week. Some EU paperwork first. https://t.co/VZeGY2yXCl

[Matt Wittmann @mcmcescher]

@thmsmlr Don't use airpods? 😏

[Matt Wittmann @mcmcescher]

@GabriellaG439 Excellent talk, thank you!

[Yutong Zhao @proteneer]

Methods pre-print from our team at @Relay_Tx: A local resampling trick for focused molecular dynamics.

Josh Fass (@maxentile) will be talking about the work on Thursday at the Free Energy Workshop in Cambridge!

https://t.co/GFwunRetAE

[Matt Wittmann @mcmcescher]

@sedatesnail Kind of want to run PageRank on the big grid now

[Matt Wittmann @mcmcescher]

@sedatesnail Would have been funnier if you'd used a fake artist as an example instead of Barb Ferbgerbus

[Matt Wittmann @mcmcescher]

@cdsmithus It all just works out if you take A -> |A|, do some algebra, and erase the |..| at the end! 😛 (would like to understand the real reason too)

[Matt Wittmann @mcmcescher]

@BayesForDays There would be a satisfying symmetry if friend => ex-Catholic and ex => Catholic

[Matt Wittmann @mcmcescher]

@BayesForDays ♥️

[Matt Wittmann @mcmcescher]

I've found today's https://t.co/XAgt7jO9hy function in 1 trie(s)!

[Matt Wittmann @mcmcescher]

@sedatesnail +1 amounts in instructions. Or even just numbered / color-coded references to the ingredients list

[Matt Wittmann @mcmcescher]

For Haskell peeps, really enjoying "Algebras for Weighted Search" by Oisin Kidney and Wu. Mind blowing! 🤯

[Matt Wittmann @mcmcescher]

@kerckhove_ts REPLay

[Matt Wittmann @mcmcescher]

@tscholak @JuliusMarozas @gsoc @hasktorch Congratulations!

[Matt Wittmann @mcmcescher]

@Eddie_Santos_3 Hmm yeah that does seem in the right direction. I'm thinking more about just translating type definitions directly from language A to B, although it would be cool to also generate codecs for a particular serialization format

[Matt Wittmann @mcmcescher]

Is there anything like "pandoc for types" that can translate ADT definitions between many different languages via some internal representation?

[Matt Wittmann @mcmcescher]

type Boool = Maybe Bool
type Booool = (Bool, Bool)
type Boooool = Maybe (Bool, Bool)

[Matt Wittmann @mcmcescher]

@chasely @remilouf @jethroksy (creator of org-roam) also has a nice example of this: https://t.co/XnUBTFfKDY

[Matt Wittmann @mcmcescher]

@tikhonjelvis @hillelogram @mitchellsalad @deech Interesting, do you have in mind something like Spark's Dataset API or Frameless (e.g. outer join result represented like (Maybe a, Maybe b)), or more like language built-ins?