Mithil Vakde

Trying to solve sample efficiency

prev: Engineering physics @iitbombay '23

[Mithil Vakde @evilmathkid]

I got 37% on ARC-AGI-2 Kaggle without LLMs!

100M looped transformer trained from scratch

- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%

thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!

Link: https://t.co/iDBUrHhmdI
(writeup for the paper track)

[Mithil Vakde @evilmathkid]

@iamrahulmeka @arcprize @fchollet @mikeknoop Thanks Rahul!

[Mithil Vakde @evilmathkid]

I got 37% on ARC-AGI-2 Kaggle without LLMs!

100M looped transformer trained from scratch

- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%

thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!

Link: https://t.co/iDBUrHhmdI
(writeup for the paper track)

[Mithil Vakde @evilmathkid]

woah

[will depue @willdepue]

I have a major result to share.

I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.

This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.

Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.

I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.

I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.

It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.

These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.

I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at https://t.co/YOtzND0Jqk.

There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.

[Mithil Vakde @evilmathkid]

@yacinelearning @arcprize @fchollet @mikeknoop i think a lot more can be done with looping!

[Mithil Vakde @evilmathkid]

@misbahspeaks @arcprize @fchollet @mikeknoop Thanks Misbah!

[Mithil Vakde @evilmathkid]

@NirantK @arcprize @fchollet @mikeknoop Thanks Nirant!

[roon @tszzl]

we should not like a future where there are 1 to 2 AIs and the rest are illegal. we should want a cambrian explosion of the types and possibilities of minds, experiencing new joys and thinking new thoughts. ensuring that this is compatible with human flourishing is only step 1

[Mithil Vakde @evilmathkid]

@datavorous_ @arcprize @fchollet @mikeknoop shhh havent yet launched it

[Anshul Bhide @anshulbhide]

LLMs require very large amounts of data and arguably the path to ASI runs through a different model architecture that’s more data sparse. Amazing what @evilmathkid has done with compute constraints on ARC-AGI-2.

cc @fchollet

[Mithil Vakde @evilmathkid]

@willdepue @arcprize @fchollet @mikeknoop thanks will!

[Mithil Vakde @evilmathkid]

@PariharCodes @arcprize @fchollet @mikeknoop ??

[Mithil Vakde @evilmathkid]

@shyamalanadkat @arcprize @fchollet @mikeknoop thanks shyamal !

[Mithil Vakde @evilmathkid]

@BhatiGarvin @arcprize @fchollet @mikeknoop Estimate it took 6 hours on Kaggle (4xL4s)

Training should take about 5hrs
Decode less than an hour

kaggle had really long queues, so cant know for sure

[Mithil Vakde @evilmathkid]

@anindyadeeps haha thanks!

[Mithil Vakde @evilmathkid]

@xennygrimmato_ @arcprize @fchollet @mikeknoop thanks!

[Anindyadeep @anindyadeeps]

And Mithil strikes again, this time casually beating opus

[Mithil Vakde @evilmathkid]

@bryanmusuku @arcprize @fchollet @mikeknoop working on this :)

[Mithil Vakde @evilmathkid]

@ram_devv @arcprize @fchollet @mikeknoop wdym?

alternative attention impl?

[Mithil Vakde @evilmathkid]

@AashaySachdeva @arcprize @fchollet @mikeknoop its extraordinary how well it works

i had basically given up, looping was a hail mary and boom it just worked

[N8 Programs ⏹️🤝⬆️ @N8Programs]

THE GOAT (of resource efficient arc impls) RETURNS

[Mithil Vakde @evilmathkid]

@82deutschmark @arcprize @fchollet @mikeknoop Haha I sweating bullets because of the submission problems on Kaggle