Timor Averbuch

Beautiful is better than ugly

[Timor Averbuch @timorchik]

@beyarkay @dhruv31415 What does that even mean? Why is higher version number better? In what world is Gemma better than anything

[Timor Averbuch @timorchik]

@PyTorch @RedHat @Meta @vllm_project Cool! Do you support interpreting Helion kernel code directly for debugging? e.g. TRITON_INTERPRET=1 in Triton

[Timor Averbuch @timorchik]

Wall Attention is now available in FLA! 🔥

https://t.co/39TnLKnCd3

Thanks @yzhang_cs, @SonglinYang4, and @zhiyuanli_.

[Tilde @tilderesearch]

Introducing Wall Attention. Diagonal forget gates enable RoPE-free attention with exceptional length generalization.

Wall outperforms the dominant method RoPE and sophisticated data-dependent methods like Forgetting Attention (FoX). We trained models with Wall on 4k sequence length and they generalized without further training to 200k+ tokens.

Wall generalizes diagonal forget gates from linear RNNs (KDA, RWKV 7, GLA) to softmax attention through a principled induced action framework. It enables transformers to selectively remember or forget per-channel within the attention head, dramatically boosting expressivity.

Wall is production-ready. Wall retains the parallel structure of vanilla attention, is compatible with GQA & MLA, and we open-source reference Triton kernels for training and decoding. Our WallDecode kernel achieves SOTA-level decode throughput.

Continual learning over long-context is fundamentally about selective forgetting → and Wall attention is all about selective forgetting.

[Timor Averbuch @timorchik]

@yzhang_cs @tilderesearch @SonglinYang4 PR is live!
https://t.co/39TnLKnCd3

[Timor Averbuch @timorchik]

This was a fun one to work on!

Some etymology: We started this project by interpolating existing gating methods across both exponential and linear kernels, and ended up with a framework that unifies FoX, PaTH, and Wall as special cases. A gate is just an opening in a wall, and Wall is what you get when you generalize gates all the way!

P.S. @dhruv31415 blessed this post with beautiful interactive diagrams that are reason enough to visit the site

[Tilde @tilderesearch]

Introducing Wall Attention. Diagonal forget gates enable RoPE-free attention with exceptional length generalization.

Wall outperforms the dominant method RoPE and sophisticated data-dependent methods like Forgetting Attention (FoX). We trained models with Wall on 4k sequence length and they generalized without further training to 200k+ tokens.

Wall generalizes diagonal forget gates from linear RNNs (KDA, RWKV 7, GLA) to softmax attention through a principled induced action framework. It enables transformers to selectively remember or forget per-channel within the attention head, dramatically boosting expressivity.

Wall is production-ready. Wall retains the parallel structure of vanilla attention, is compatible with GQA & MLA, and we open-source reference Triton kernels for training and decoding. Our WallDecode kernel achieves SOTA-level decode throughput.

Continual learning over long-context is fundamentally about selective forgetting → and Wall attention is all about selective forgetting.

[Timor Averbuch @timorchik]

@AntLingAGI @marterrz Did you release the Triton kernel?

[Timor Averbuch @timorchik]

In general, MFU is a very confusing and inconsistent metric. I've found that some reports omit the optimizer step entirely when reporting it, and in a mixed-precision setting it makes even less sense since the baseline denominator is ambiguous. MFU is really only useful for apples to apples comparison between identical kernels like two FP8 GEMMs.

[Timor Averbuch @timorchik]

@maciejpioro @tilderesearch It's in Section 5 under "Aurora 1.1B Pretraining," with more details in Appendix B.

We've also published our 100B token checkpoint on Hugging Face. Check it out here: https://t.co/z5WvxVnXDE

[Timor Averbuch @timorchik]

@AJacksonHole @zephyr_z9 We are currently working on scaling this approach. The 1B results came as a surprise after we decided to evaluate a checkpoint we had originally trained to verify correctness. Just 100B tokens, no post training, no tuning, no architectural optimizations.

[Timor Averbuch @timorchik]

Once again, principled research beats guesswork 😎

[Tilde @tilderesearch]

Introducing Aurora, a new optimizer for training frontier-scale models.

We train Aurora-1.1B, which achieves 100x data efficiency on open-source internet data. Despite having 25% fewer parameters, 2 orders of magnitude fewer training tokens, and using fully open-source internet-only data, Aurora matches Qwen3-1.7B on several benchmarks.

Aurora was developed after identifying a major failure mode that can occur under Muon, an increasingly popular optimizer that has shown strong gains over Adam(W). We find that Muon can cause a huge percentage of neurons to effectively die early in training, reducing effective network capacity so that many parameters no longer meaningfully contribute to network outputs.

By redistributing update energy more uniformly across neurons while preserving Muon’s stability properties, Aurora prevents neuron death and recovers substantial model capacity.

What makes this work especially exciting is that it points toward a broader direction for ML research: better optimizers may not come purely from elegant mathematical abstractions, but from understanding and addressing the concrete dynamics and pathologies that emerge inside real training systems.

[Timor Averbuch @timorchik]

@JadeYi25 @tilderesearch Stay tuned for the release tomorrow! 😉

[Timor Averbuch @timorchik]

@_lewtun @tilderesearch Definitely, happy to facilitate!

[Timor Averbuch @timorchik]

Distillation (on-policy in particular), is one of the highest-leverage tools small teams have for training frontier-level models efficiently.

By open-sourcing Nitrobrew, we hope to make scalable distillation more accessible and help more teams push model quality without needing frontier-scale infrastructure.

[Tilde @tilderesearch]

Distillation (especially on-policy) has become a pivotal component of the post-training stack.

☕ To dramatically accelerate distillation at scale, we open-source Nitrobrew, a communication-efficient, fused strategy for logit distillation. It’s built for both on- and off-policy distillation with:

100x faster loss computation
50% peak memory savings
3x faster on-policy distillation
and more!

A 🧵 (1/8)

[Timor Averbuch @timorchik]

Come meet us at NeurIPS

[Tilde @tilderesearch]

Tilde is heading to NeurIPS! A few of us will be around and would love to meet others working on pretraining + model architecture.

We’re also hosting a dinner on Dec 3. DM us or our Chief of Staff @gitipahwa if you want to join!

[Timor Averbuch @timorchik]

# [B, N, H, D]

[⏧ @syntacrobat]

literally how do ML people survive without lifting tensor dimensions into the type system? isn't that like the number one thing youd immediately want

[Tilde @tilderesearch]

Modern optimizers can struggle with unstable training. Building off of Manifold Muon, we explore more lenient mechanisms for constraining the geometry of a neural network's weights directly through their Gram matrix 🧠

A 🧵… ~1/6~

[Timor Averbuch @timorchik]

Incredible team doing incredible work

[Origin @origin_bio]

Introducing Axis: the first AI model that generates regulatory DNA elements and predicts their function.

Gene therapies suffer from poor efficacy, toxicity & specificity. Models like Axis can help overcome such risks.

Axis beats Google DeepMind's AlphaGenome at predicting regulatory element binding activity by 6.7%.

[Tilde @tilderesearch]

We’re excited to announce that Tilde completed an $8M seed round earlier this year, led by Khosla Ventures.

Understanding model intelligence is the most important problem in the world, and the key to actualizing the promise that ASI can offer.

🧵 A thread on our approach: