Deven Pietrzak

MIT math, Polish national baseball team, social physics lab.

[Deven Pietrzak @DevenPzak]

Excited to have some of my pretraining results published!

1. We’re announcing ANVIL III, the new pretraining optimizer that builds on ANVIL I and ANVIL II (the optimizers I created for the nanoGPT speedrun). ANVIL III takes a new approach, and is much more suited to large-scale models and overtraining, where the advantage grows as we scale up. It features several changes that @Everule2 and I made after the publication of ANVIL II in the nanoGPT world record. ANVIL III, in our internal testing, shaved even more time off the nanoGPT speedrun, but we are not willing to publish the IP at this moment, as it is an optimizer that can make a massive impact on the frontier of pretraining. A SNR distribution curve of ANVIL III compared to muon is attached. ANVIL III also has remarkably consistent optimal parameters across model scales and overtraining regimes, making it exceptionally well-suited to frontier pretraining.

2. Over the past 4 days, I used ANVIL III and architectural changes I had come up with back in July, to train a LLM primarily on web data. These combined to create Feather-1.7B, a 1.7-billion parameter base (only pretrained) model, trained entirely on 200B tokens of public datasets (decontaminated thoroughly across all benchmark test sets). At 180x less compute, it outperforms Qwen3-1.7B, the frontier base model at its size, on math, while matching it on reasoning. It is competitive with Qwen3-4B on math, despite being trained with 414x less training compute.

We are unfortunately compute-constrained, so we did zero parameter tuning at 1.7B scale and did only one full training run of this model. Despite likely being far from its optimized frontier, we still outperform Qwen3-1.7B and compete with Qwen3-4B on math benchmarks, without losing any language/reasoning capabilities.

3. Last but not least, the nanoGPT speedrun world record I created and submitted, taking the record from 73.889 to 39.914 seconds. This record marks a 1.851x speedup, the largest ever and larger than the last 45 world records combined.

Really excited to continue the work, and looking forward to some of the other insane results we will be publishing soon @hyperstition_cc!

Link to pretraining blog: https://t.co/PyfvnSSwVC

[Hyperstition @hyperstition_cc]

Today, we release the first series of our findings.

1 - We cut pretraining costs by 62% at frontier scale (8x Chinchilla), and we further cut all AI inference costs by 30% (Faster decode).

2 - Introducing our model ‘Feather’, which decisively overtakes Qwen 3 as the leading frontier 1.7B math model, using 180x fewer total training tokens, even matching Qwen 4B at potent benchmarks in math.

3 - With Anvil II, our state of the art LLM Optimiser, we record the largest pre-training efficiency jump on the NanoGPT Speedrun, leaping the previous by 34seconds - Our record singularly, is a greater percentage drop than the past 45 World Records combined.

The website : https://t.co/fG1TZecjIE

[Deven Pietrzak @DevenPzak]

Neither.

The nanoGPT run is entirely separate, as that one used the deprecated ANVIL II, and its gain is both from the whole submission and cannot be perfectly extrapolated.

We have measured ANVIL III vs tuned Muon at 124M-1.2B parameters, up to 40B tokens. The loss gap between ANVIL III and Muon stays about constant (even slightly increasing). At chinchilla optimal at 1.2B, the gap is ~29% fewer tokens. Interpolating that to 8x Chinchilla, as shown and calculated in our appendices, leaves us with a ~50% reduction. Adding in the architectural changes, we reach 62%.

The controlled ablations aren’t in the blog post, but we took the pessimistic number over the ~70% that you’d get from a naive extrapolation.

[Deven Pietrzak @DevenPzak]

This was my record. Thanks for the post, Ryan! Would love to give early access to some of our other extensions and findings in private.

One interesting fact is that autoresearch played almost no role in my work. In fact, over the five days I worked on this, I had a cluster of 4-6 pods (depending on usage by others), full-time. At all times that I wasn't actively working, I had agents using every pod I had access to, trying to improve my record. On net, over the full five days, they spent dozens of hours trying to hill-climb this speedrun, and they made less than a second of progress.

While I do think that the fast feedback loops and clear goal makes it more approachable for agents to improve on this benchmark, it seems they still struggle with it. No matter how hard I tried to steer them away from it at the start, the agents always ended up doing useless parameter tuning or chasing down clearly unfruitful paths. My results seemed similar to those from this METR blog: https://t.co/ZtKPp2w7nT

I also want to clear something up. Many people seem to take issue with the large Bigram/Trigram embedding table I implemented.

First of all, it is only worth 7 seconds out of the 34 seconds.

I feel I should clarify that this table already existed, at a size of 290M parameters, well over twice the size of the transformer. And the reason that it was only 290M parameters was not because nobody wanted to try increasing it; it was that it would have made the run slower. The Bigram embedding table, significantly larger than the transformer itself, was accepted in 28 straight records.

The old code kept copies of the table on every GPU, wrote and zeroed a table-sized gradient every single step, and wrote the table back to every GPU after every step. This made the cost linear in table size but the gains roughly logarithmic; 290M was the crossing point.

I sharded the table across 8 GPUs, pulled only the rows the current batch’s tokens actually hashed to, sent gradients back as a segment-sum rather than table-sized tensor, and ran Adam only on the touched rows. Then the optimizer cost scaled with rows used per step, making it almost invariant to table size (1/4 size embedding table, 16B instead of 65B, only saved 0.17 seconds).

In effect, the size of the Bigram table is a tuned parameter. In the past, the optimal size of the table was 290M params for the reasons I said above. Through my engineering breakthroughs, the table became nearly free to add to. This made the optimal tuning for the table size parameter significantly larger, in the order of 65 billion. The only thing I did was make one part of a speedrun much more efficient, and retuned the size / composition based on that efficiency.

Even if I kept the table the exact same size and only kept the engineering breakthroughs, the engineering speedups alone would have cut 1.7 seconds off the record.

[Deven Pietrzak @DevenPzak]

@RyanGreenblatt Yes, almost all of the implementation was done with claude code. It allowed me to focus on ideation, especially when I could send out agents to do kernel builds and similar stuff in the background while I spent more time on strategy.

[Deven Pietrzak @DevenPzak]

@vvvincent_c @tmkadamcz I had several people question this, so just to test, I removed it and added steps until I got back to the <3.28 bar, with some room. It ended up adding 7 seconds, validating the ablation table!

[Deven Pietrzak @DevenPzak]

This was my record. Thanks for the post!

It seems a few people don’t like the large bigram/trigram hashed embedding table. I’d like to clear up a couple misconceptions about it:

1. It’s only worth 7 seconds of the 34-second record. Published in the ablation, and rerun myself (finished version with loss <3.28 and no embedding table growth ended up at 46.9 seconds).

2. People think nobody had done it because it was too many parameters / against the spirit of nanoGPT. In fact, it’s the opposite. The table was already 290M parameters, and the reason it hadn’t grown was because it would have made the time worse.

The old code kept copies of the table on every GPU, wrote and zeroed a table-sized gradient every single step, and wrote the table back to every GPU after every step. This made the cost linear in table size but the gains roughly logarithmic; 290M was the crossing point.

I shared the table across 8 GPUs, pulled only the rows the current batch’s tokens actually hashed to, sent gradients back as a segment-sum rather than table-sized tensor, and ran Adam only on the touched rows. Then the optimizer cost scaled with rows used per step, making it almost invariant to table size (1/4 size embedding table, 16B instead of 65B, only saved 0.17 seconds). This let me push to 224x and add a trigram channel.

Even if I didn’t expand it, just from the engineering described in the last paragraph, I would have saved over one second from this change alone. The real reason no one had expanded the embedding table was not because of the table itself but rather because it would have been too slow. The table was already more than double the model size, so we were far past the realm of small, scaling-intended changes to the bigram table.

Either way, I appreciate the post, and this was a fun project to work on! Happy to answer any questions anyone may have about the record.

[Deven Pietrzak @DevenPzak]

Fun side project while I’m in Europe! Proud to have the biggest record since the early days of the challenge. (Pending official verification, for now)

[Social Physics Lab @socialphysics_]

The Nano-GPT speedrun has been, in effect, the most potent contextualization of pre-training efficiency. On its first iteration in 2024, @karpathy set the baseline at 45 minutes. The ability to shrink that to under a minute, in a tidy couple years, stands as a testament to the pace of LLM advancement.

Our public implementation beats the current Nano-GPT speedrun record by 8.7 seconds, bringing the new world record to 65.87 seconds - the single largest absolute jump in over 18 months.

We have insofar redacted the capabilities that substantially move it to under the 60 second threshold for reasons proprietary to SPL.

This, though nimble, marks as a singular, among a panoply of findings SPL keenly expects to release over the coming weeks.

Kudos to @DevenPietrzak who individually theorized and implemented it within a few short nights, while simultaneously pitching and taking to the European finals, Poland’s premiere baseball squad.

Here’s the github PR : https://t.co/yZsnlfL635