Albert Gu

assistant prof @mldcmu. chief scientist @cartesia. leading the ssm revolution.

[Albert Gu @_albertgu]

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

[Albert Gu @_albertgu]

i mean i'll take it (but really our research team is incredible)

[shan @babyshan445]

A few Nvidia interns and I officially ranked tech companies in 2026...

[Albert Gu @_albertgu]

@rdesh26 indeed bytes themselves aren't raw and UTF-8 is quite arbitrary. Hindi in particular has some interesting behavior from our latest internal H-Net because of weirdness in its encodings

@karpathy and others have also espoused training on images and pixels, makes a lot of sense!

[Albert Gu @_albertgu]

it's hilarious how many have taken the bitter lesson to mean just scaling up transformers. it's about scaling, but innovations in algorithmic primitives can lead to much more efficient scaling

[Richard Sutton @RichardSSutton]

Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO

[Neehal Tumma @ntumm120]

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it?

It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯

Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

[Thomas Wolf @Thom_Wolf]

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

[Albert Gu @_albertgu]

new open weights model from a dark horse

[Ricursive Intelligence @RicursiveAI]

Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.

Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.

Learn more: https://t.co/Pe04ZKhkE9

[Albert Gu @_albertgu]

@giffmana 😂 glad to hear any wisdom drops

[Albert Gu @_albertgu]

byte-level models are better under multi epoching (e.g. training K times over 1/K of the data). another connection between learning from raw data <-> improved data efficiency

[Wenhao Chai @wenhaocha1]

n-gram dominates recent pre-training speedruns, however, it totally sucks under multi-epoch setup. small vocabs survive.

larger tokenizers suffer more multi-epoch regret, the extra loss from repeating data instead of training on unique data.

Blog: https://t.co/nN9hJGCtvq

[Albert Gu @_albertgu]

in a constantly changing model landscape, faster adaptation is the biggest advantage

[Simran Arora @simran_s_arora]

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨

Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.

Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.

From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

[Albert Gu @_albertgu]

@TomLimi great set of directions! I agree these are all important :D

[Albert Gu @_albertgu]

@eaplatanios yep we tried this in the original paper (last appendix). it doesn't work so well because of subtleties, in particular BPE has "causal leakage". there have been other related works like Bolmo that try distilling

[Albert Gu @_albertgu]

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

[Albert Gu @_albertgu]

@dadchords there's definitely some nuances; bytes are themselves arbitrary and "tokenized" (e.g. utf-8 encoded bytes) and we glossed over all of that discussion. actually using pixel bitmaps is definitely an interesting idea that's come up before and I think there is some potential there

[Albert Gu @_albertgu]

Our opening animation and blog didn't fully show the interesting dynamics for languages like CJK that have multi-byte characters.

In Chinese most characters take up 3 bytes, and when learning end-to-end from bytes it's not even trivial to get the model to chunk these into characters. Early in training this H-Net incorrectly sub-divides characters, but it eventually figures it out. Actually this was a prominent issue in the original H-Net - and even BPE doesn't get it right on rare characters. The blog has some examples!

[Albert Gu @_albertgu]

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

[Albert Gu @_albertgu]

@OllieProudfoot You can try but I don't think it'll be easy 😄

[Albert Gu @_albertgu]

this is an abuse of technology 😭

[llm_enjoyer @LLMenjoyer]

one thing @_albertgu won't tell you is how well hnet tokenizes bad apple vs bpe.
suspicious! what else is he hiding???

[Albert Gu @_albertgu]

@Samrn888 I just looked at this, this is a really nice writeup very cool! It's another nice study on how semantically meaningful tokenization matters

[Albert Gu @_albertgu]

@hmdolatabadi it does affect MFU and requires a lot of engineering to get this to train efficiently, but it's doable!

[Albert Gu @_albertgu]

yes utf8 is an arbitrary tokenizer too; we're definitely overloading the word "tokenize" a bit here. I've also thought about how other scripts might have more natural "atomic units" they're composed of and it could be interesting to work over those instead. I'd expect the models might get better. but it would require a lot of engineering in a different way as well

[Albert Gu @_albertgu]

@OllieProudfoot I wasn't aware there existed some previous checkpoints on HF (can't remember if we actually uploaded some lol). but in-context tokenization is something that does require more scale; it emerges only after hundreds Bs of tokens in training

[Albert Gu @_albertgu]

yeah, one of the downsides of this is a very increased hyperparameter surface area. getting the parameter distribution across stages and the compression ratios is very nontrivial already. we have various H-Net ladders with different properties; the one shown in the post was one that was very simple and tried to be matched to the BPE model without doing these sweeps, but there can be more optimal configurations