Kilian Haefeli

pretraining @cohere & master @ETH

[Toni Kroos @ToniKroos]

Football won

[Rodrigo Casado Noguerales @rodcasnog]

1/ If you have been reading about discrete diffusion, you might have asked yourself the following question:

What does a discrete diffusion model actually learn?
A denoiser? A concrete score? A bridge plug-in?

The literature treats them as interchangeable. They are not! 🧵

[Fabian Schaipp @FSchaipp]

How to scale the batch size in LLM pretraining?

New paper on a scaling law that splits token budget D into training steps and batch size.

💺 https://t.co/eXTcIjKyqY

[Kimbo @kimbochen]

You know what model won’t reject your requests?
Cohere North Mini Code

[Cohere @cohere]

Introducing Cohere's first open-source coding model: North Mini Code

Small & efficient, designed for agentic performance and built for community input.

[driss guessous @drisspg]

To hell with big TMA long live ld/st

[Kilian Haefeli @khshind]

@eliebakouch @kimbochen All our models do yes :)

[Cohere @cohere]

Introducing: Cohere Command A+

We’ve created our most powerful LLM yet, optimized it to run on as little hardware as possible, and released it open-source for all.

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

just your friendly reminder to throw away any RL paper that only tests their method on Qwen models :)

[Kilian Haefeli @khshind]

@dwarkesh_sp @SemiAnalysis_ @dylan522p sure its his intern?

[Ekagra Ranjan @EkagraRanjan]

Ever wondered how Speculative Decoding interacts with production MoE models?

Conventional wisdom: MoE + speculative decoding = too many experts to load, gains disappear.

Reality: MoE amplifies speculative decoding.

Checkout Cohere Blogpost: https://t.co/0iqZpYfXIN

[Anselm Levskaya @anselmlevskaya]

I work on the JAX team. If you're new to the field ignore this bait. The things you should focus on are understanding the math and how to program accelerators - really master your hardware and your methods. We try to make a great tool but don't obsess over tools early.

[Nicholas Boffi @nmboffi]

🤯 big update to our flow map language models paper!

we believe this is the future of non-autoregressive text generation.

read about it in the blog: https://t.co/DfBXrYmJc8

full details in the paper: https://t.co/coiNXj4ucC

we introduce a new class of continuous flow-based language models and distill them into their corresponding flow map for one-step text generation.

we beat all discrete diffusion baselines at ~8x speed!

v2 gives a complete theory of the flow map over discrete data, with three equivalent ways to learn it (semigroup, lagrangian, eulerian). it turns out you can train these with cross-entropy objectives that look very similar to standard discrete diffusion — but without the factorization error that kills discrete methods at few steps.

beyond improving results across the board, we showcase properties that are unique to continuous flows. in particular, inference-time steering and guidance become straightforward. autoguidance brings generative perplexity down to 51.6 on LM1B, while discrete baselines completely collapse at the same guidance scale.

we also show reward-guided generation for steering topic, sentiment, grammaticality, and safety at inference time — and it works even at 1-2 steps with our flow map model. simple, well-understood techniques from continuous flows just work incredibly well in practice for language.

we’re extremely excited about the future of this class of models.

stay tuned for results on scaling, reasoning, and reinforcement learning-based fine-tuning. 🚀

[Kilian Haefeli @khshind]

Awesome deepdive by kimbo!!

[Kimbo @kimbochen]

Previously when writing the NVIDIA Tensor Core evolution article, I realized how scarce the info about Blackwell micro architecture is
So I set off doing this to bridge the gap.
Super thankful for my colleagues and collaborators
Thank you @snowclipsed for reviewing too

[Kilian Haefeli @khshind]

@giffmana @LLMenjoyer I recently found out heidi was made by the ghibli founders in jp and never officially aired in swiss tv

[Kilian Haefeli @khshind]

@Laz4rz Here we are

[Kilian Haefeli @khshind]

@prime_cai Absolutely

[Artem Lukoianov @ottogin1]

Join us at @CVPR in Denver for a full-day tutorial about Analytic Understanding of Diffusion Models.

The training objective of diffusion models has a closed-form solution -- yet it only memorizes. How do real models generalize? We'll unpack this paradox and the emerging analytical theory behind it.

@yuancy @CScarvelis @MasonKamb @WangBinxu @vincesitzmann @JustinMSolomon @SuryaGanguli

[Antonio Orvieto @orvieto_antonio]

Optimization theory for adaptive methods actually predicts most of what we know about hyperparameter scaling in LLM pretraining, and suggests new strategies as well. We did a deep dive here.

[Emiel Hoogeboom @emiel_hoogeboom]

You may think discrete distillation is fundamentally flawed, you are (surprisingly) wrong. 🤯

Meet Discrete Moment Distillation (D-MMD). It is a new method that brings fast, few-step sampling to discrete diffusion models! 🧵👇