Diego de las Casas

Working on LLMs at @MistralAI
Past: @GoogleDeepMind
🇧🇷 in 🇬🇧

[Pierre Stock @PierreStock]

ML4 is now open-weight! Frontier capabilities on code agents & cyber, way more efficient and compact than its Preview version three weeks ago (!) and cherry on the cake: technical report telling its epic backstory. Check it out: https://t.co/TOOfN9Pob2

[Diego de las Casas @diegolascasas]

I don't think this is the most likely path but I can see a world where small mem-efficent models are reliable enough to be smart assistants and large batched models are still industrial drivers. PII handled locally, hard problems escalated to cloud/enterprise.

[morgan — @morqon]

sam: local inference is a terrible allocation of scarce resources

“a lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre

but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time

you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment

somehow many people's intuition is the opposite of that”

[Diego de las Casas @diegolascasas]

@MaxMynter The UK will be an AI powerhouse because it already mastered the art of turning churches into everything else

[Diego de las Casas @diegolascasas]

Every European datacenter should be eurocoded: church, train station, hospital, winery, cigarette factory...

[Diego de las Casas @diegolascasas]

@AlbertQJiang Bit too French, we could do it with some green hydrogen, but I take it

[Diego de las Casas @diegolascasas]

You may not like it but this is what peak performance looks like

[Guillaume Lample @ NeurIPS 2024 @GuillaumeLample]

ML4 was trained on 3,800 NVIDIA Grace Blackwell GPUs in our European datacenters -- our cluster in Bruyères-le-Châtel built using our Series B fundraise.
We are investing heavily in infrastructure, with our Series C and D clusters coming online soon. This additional compute will let us train longer, pursue more ambitious post-training, and iterate faster.
We expect large and rapid improvements in the weeks and months ahead.
2/n

[Diego de las Casas @diegolascasas]

@recurseparadox But the thing is the difference between bf16 and w4a4 right?
A 1Gb model would be 125M (bf16) or 250M (w4a4) but the latter would have the perf of a ~2.5Gb b16 (312M) model with same training budget.

[Guillaume Lample @ NeurIPS 2024 @GuillaumeLample]

Mistral Medium 3.1 is 2nd on LMArena without style control.
Very proud of the @MistralAI team !

[Val @onetwoval]

Introducing Magistral, a model that is good at counting, among many other things

Here it is solving a little puzzle with all tools disabled
(video in real time)

[Albert Jiang @AlbertQJiang]

Glad to share what we've been working for the past 4 months. We've got some sweet RL stack and two nice reasoning models. Try it out at https://t.co/yQlphmWTap, select think/pure thinking.

[Sagar Vaze @Sagar_Vaze]

Our first LLM coding agent: Devstral.

It's the SoTA OSS model on SWE-Bench Verified by a large margin, while being only 24B.

It's still a research preview specific to SWE tasks. For regular code completion Codestral might still be the best bet. Let us know your feedback!

[Diego de las Casas @diegolascasas]

@dchaplot Congrats!

[Emmanuel Macron @EmmanuelMacron]

Vive Le Chat ! 🇫🇷

[Val @onetwoval]

No this video is not sped up, genuinely mind blowing

And yes this is available to all users right now.

[Diego de las Casas @diegolascasas]

you know what to do

[Mistral AI @MistralAI]

Introducing Small 3, our most efficient and versatile model yet! Pre-trained and instructed version, Apache 2.0, 24B, 81% MMLU, 150 tok/s. No synthetic data so great base for anything reasoning - happy building!

https://t.co/8WfgzuuwRk

[Diego de las Casas @diegolascasas]

@GolinoHudson They likely don't have a large number of collocated GPUs (their networking is bad and can't scale). They say it's 2k h800s for 2 months, but that's just for that training run. They probably used a lot more for experiments, data generation, etc.

[Diego de las Casas @diegolascasas]

@Ar_Douillard Don't think it's a matter of noise, you'd need to look at how correlated are benchmarks with things that matter (construct/predictive validity)

[Diego de las Casas @diegolascasas]

What a year!

[Sophia Yang @sophiamyang]

Extremely proud of all the models and products we’ve shipped at @MistralAI this year! So many more to come in 2025. Stay tuned!

🌟Highlights from 2024:
- Feb: Mistral Large, Mistral Small, Le Chat
- Apr: Mixtral 8x22B
- May: Codestral
- Jun: Fine-tuning API
- Jul: Mistral Large 2, Mathstral, Codestral Mamba, Mistral NeMo
- Aug: Mistral Agents
- Sep: Pixtral 12B, Mistral Small v24.09, free API tier
- Oct: Ministral 3B & 8B
- Nov: Pixtral Large, Moderation API & model, batch API; Le Chat new powers - search, vision, pdf understanding, image generation, Canvas