Hyperstition

For too long has coordination been the Achilles’ heel to human progress. DMs open for collabs.

[Hyperstition @hyperstition_cc]

We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency.

Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold).

All of this in a single kernel launch.

For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations.

Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies).

Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it.

Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :)

GitHub repo: https://t.co/sfPt4S4BJN

NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs).

1/9

We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash.

Compared with NVIDIA’s published vLLM results, Hyperkernel delivers:

• 36.7% higher per-user generation speed at concurrency 1.

• 28.2% higher per-user speed while serving two users, versus NVIDIA serving one.

• 15% higher per user speed while serving four users, versus NVIDIA serving one.

• 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s.

• 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result.

• 36% faster median per-request decode speed.

[Hyperstition @hyperstition_cc]

Attention patterns vary more across innit seeds than they do across training!

[Hyperstition @hyperstition_cc]

Today we report a startling pretraining finding:

Attention circuits STOP meaningfully evolving much sooner than we expect.

In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it.

The setup:

We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other.

The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen.

NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent.

Some differences in the loss are attributable to these different attention patterns.

This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts.

This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck.

A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting.

Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as

Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ

i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is

∂L/∂sᵢₖ = −aᵢₖAᵢₖ

A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes.

In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where

Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ.

Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel.

The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them.

This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners.

The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ):

Fᵢ = Covₐᵢ(k) / d

uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ

g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂)

Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain.

Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession.

Here’s the complete blog: https://t.co/bLPjndoxWE

We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability.

Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.

[Hyperstition @hyperstition_cc]

The attention scores span far more orders of magnitude than the advantages.

[Hyperstition @hyperstition_cc]

@alon_rag you are exactly right. watch out for our drop later next week ;)

[Hyperstition @hyperstition_cc]

Why solving hang detection matters for pretraining efficiency:

NCCL takes minutes to detect hardware faults, and can require manual intervention in the case of a silent hang. Large scale pretraining runs have faults every few hours, and teams lose 10-15% on wall time to fault overhead.

Our heartbeat protocols detect NVLink, InfiniBand and GPU faults in seconds, and reload the minimal affected model state from healthy peers. Then, the megakernel is regenerated to preserve MFU in the degraded cluster. This requires recalculating optimal sharding of experts and tensors, as well as assigning less work to degraded DP groups.

[Hyperstition @hyperstition_cc]

3/n

We have the most complete pretraining megakernel with hardware fault detection and replanning.

1. We achieve an out of the box 31% MFU on standard mixed precision (BF16 and FP32) pretraining of a 30B MoE Qwen architecture on one node and 30% MFU on data parallel pretraining over multiple nodes.

2. We have tensor parallelism, expert parallelism and data parallelism.

3. Hyperkernel automatically generates efficient megakernels that fuse low intensity processes and schedule the tiles to load balance over applicable streaming multiprocessors. This generalizes across all cluster topologies.

4. Hyperkernel automatically detects individual GPU, NVLink, and InfiniBand failures within seconds. Performs live triage mid-step so as to only recompute tensors which were lost rather than naively reverting to a previous checkpoint. Autonomously rewrites the megakernel to preserve MFU on the now partially degraded cluster.

[Hyperstition @hyperstition_cc]

Robotics is more latency bound than we think.

A big reason π0 style VLAs carries a separate denoising head is that token by token decoding has been too slow to serve. https://t.co/RBZZkWzI2k

Autoregressive decoding at batch 1 is latency-bound. On a typical stack, every token has a python step overhead, a stream of kernel launches and a host sync.

Our π0-FAST megakernel runs the whole chunk in one launch: vision prefill, then the full decode loop on the GPU, with KV cache resident, sampling on device. We also have ops overlap across layers with no kernel boundaries.

94 ms, vs 243 ms for PyTorch + CUDA graphs.

Credit: @Nahomes_15

[Hyperstition @hyperstition_cc]

9/n

Our megakernel on robotics:

π₀-FAST, from Physical Intelligence @physical_int, is a vision-language-action model for dexterous robot control. Instead of generating actions with a flow-matching action expert, as π₀ does, it compresses them into discrete tokens and decodes them autoregressively.

Our megakernel ran the vision encoding prefill and generation of 60 tokens in 94 ms, while a custom reference PyTorch + CUDA graph decoding ran it in 243ms, with megakernels reducing latency by 61% at batch one in BF16.

[Jerry Tworek @MillionInt]

Megakernels are the future and incredibly cracked teams seem to be pushing the boundaries of possibility each day getting further and further.

Looking forward to seeing more from this team 😀

[Hyperstition @hyperstition_cc]

11/11

Our megakernel on a two handed manipulation robot policy:

ACT, from Stanford's ALOHA project, is a robot policy that does two-handed manipulation from a few dozen human demonstrations. Since it runs every camera frame through the control loop, inference time is the primary determinant for how fast the robot can react. Our megakernel ran it in 0.81 ms per frame versus 2.21 ms for Hugging Face LeRobot in BF16 with torch.compile, 2.7× faster, with unchanged task success.

[Hyperstition @hyperstition_cc]

We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency.

Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold).

All of this in a single kernel launch.

For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations.

Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies).

Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it.

Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :)

GitHub repo: https://t.co/sfPt4S4BJN

NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs).

1/9

We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash.

Compared with NVIDIA’s published vLLM results, Hyperkernel delivers:

• 36.7% higher per-user generation speed at concurrency 1.

• 28.2% higher per-user speed while serving two users, versus NVIDIA serving one.

• 15% higher per user speed while serving four users, versus NVIDIA serving one.

• 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s.

• 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result.

• 36% faster median per-request decode speed.

[Hyperstition @hyperstition_cc]

10/n

Our megakernel on voice models:

Moshi is a full-duplex speech-to-speech model behind real-time voice agents.

Our megakernel ran the Moshi language model and the deformer in 6.8 ms per frame versus 9.45 ms for the PyTorch reference with CUDA graphs. This is a 28% lower measured latency at batch one.

[Hyperstition @hyperstition_cc]

Training NanoGPT in 9 seconds is only a teaser for what it's redacted extensions bring. Anvil III, along with further architectural advances, 4x the effective compute output for pre-training. 600bn dollars in GPUs can now train a model 2.4 trillion in GPU's can. @DevenPzak expounds on how Anvil scales at the frontier.

[Deven Pietrzak @DevenPzak]

Since my nanoGPT record was published, the question I’ve been asked the most is, “Does it scale?”

The true answer is, while some aspects of the speedrun scale, the goal of this specific endeavor was not to scale it up. The point of the challenge is to utilize clever engineering and creative ML intuition to increase efficiency. But while nanoGPT is just about hill-climbing one benchmark, the rest of my work has been focused on scaling to the frontier.

The main projects I’ve worked on are some architectural changes and ANVIL III. ANVIL III is a significantly stronger optimizer than ANVIL II, the optimizer I used to beat the nanoGPT world record. In fact, for the record, we removed the parts of ANVIL III that scale.

ANVIL III is significantly stronger than Muon, while nearly matching its speed. For the same token budget (same compute), ANVIL III reaches final validation loss 20-28 millinats (.02-.028 nats) lower than Muon, holding at all scales from 124M to 1.2B, and different overtraining depths. As we scale up the model size and training depth, the 20-28 millinats becomes worth a larger fraction of compute. At deep overtraining (8x-Chinchilla) and 1.2B parameters, the fitted scaling curves imply that ANVIL III reaches the same loss numbers at 50% less compute. Of all the scales we tested, the gap is largest at 1.2B, staying between 26-28 millinats.

The first remarkable aspect of ANVIL is hyperparameter consistency. The optimal hyperparameters remain highly consistent across different scales and training depths. In most of our tests, varying hyperparameters rarely changed ANVIL III’s loss, or did so within noise. In fact, as shown in the attached image (from the blog’s appendices), we tuned Muon significantly more than we tuned ANVIL III. We performed over 3 times the aggregate amount of runs of Muon than ANVIL III, and at each scale our loss margin uses only the optimal parameters for both. At 1.2B, we used 51 tuning runs of Muon (including muP proxies), compared to only 7 for ANVIL. We wanted to guarantee that our improvement was not a tuning artifact, so we meticulously tuned Muon’s parameters, and had to apply far less tuning to ANVIL III. Our hypothesis is that the high SNR of ANVIL III makes it very resistant to parameter changes.

That SNR is the second remarkable aspect of ANVIL III. We generalize the Adam SNR (SNR = m̂ / (√v̂ + ε)) for each singular direction (uᵢ, vᵢ) in an update, and measure how strongly held-out gradients (G_b = gradient on held-out batches b) point along it:
SNRᵢ = mean_b(uᵢᵀ G_b vᵢ) / rms_b(uᵢᵀ G_b vᵢ)
In our tests, Muon takes a lot of steps with low SNR, but ANVIL III doesn’t, despite being a Muon variant. For example, at 10% of the training tokens into a 1.2B (approximately Chinchilla) model, 90% of singular directions in ANVIL III’s steps go downhill on held-out data, while Muon gets only 67% (random = 50%). Pretraining Blog: https://t.co/PyfvnST4La

Another recent post from Hyperstition contextualizes the importance of this SNR gap. In The Signal and the Noise, we generalize Adam (in which the update is just a per-coordinate SNR) to every direction in parameter space. Almost every direction is nearly orthogonal to the gradient history, so averaging that over all directions just collapses back into plain momentum. But the single best direction, C⁻¹ḡ, reaches SNR=1 whenever recent gradients are linearly independent, which is almost always true in LLM pretraining. By solving that with a s-by-s Gram system with recency weighting, we get a basis-free optimizer that matches AdamW at 286M parameters. While this is far from optimal, it represents one avenue for possible gains in pretraining optimization that are yet to be realized. https://t.co/8zqB3X4mXF

In another fun but mostly unrelated blog, Stuck in a Loop, we show that attention in pretraining creates feedback loops and self-reinforcement. A key’s gradient is proportional to the attention it already gets, so keys with early inflated attention receive stronger gradients, overinflating them. Breaking that loop spreads attention out, with the average entropy of attention heads goes from 3.75 to 4.69 nats (effectively 43 vs 109 tokens), but it makes the model worse. Similarly to The Signal and the Noise, this suggests that putting more weight in high-signal directions can beat spreading it out broadly. https://t.co/UKL2ACMuhb

These discoveries are just a small part of the upcoming releases from @hyperstition_cc. Can’t wait until we release more. Big things are coming!

[Deven Pietrzak @DevenPzak]

I think I just trained NanoGPT in 9.65 seconds!?

Yesterday, @hermanbrunborg used a connected longest exact match model, trained concurrently on CPU, to get the nanoGPT record from 39.9 seconds to 21.5 seconds, in an impressive open PR. By adapting their work, and generalizing the CPU model to a larger share of the tokens, I got the record down to 9.65 seconds, and submitted the PR this morning.

I built on three open PRs, in addition to my contributions: the main basis for my generalization was from @hermanbrunborg, and additional work from @nthngdy, @cyrusghane, and Daniel Monroe. Their engineering is extremely impressive.

Of course, all of our records are still subject to review. As I mentioned in my comment on Herman‘s PR (and in my own PR description), CPU usage is a gray area in nanoGPT. Their PR was the first to discover a clever way to effectively use CPU during the run, so I’m looking forward to the official reviews.

This was another very fun challenge! Particularly (potentially) breaking the 10 second barrier yesterday. So much engineering tact is yet to be explored.

Huge shoutout to the three authors of the PRs for their innovations.

@hermanbrunborg - I will also be at stanford for the weekend along with some of my cofounders, and sf for a couple days after, would love to chat more IRL, if you’d be open 🙂