Alec Dewulf

training language models @tilderesearch

[Alec Dewulf @AlecDewulf]

completely agree. in general, I think a world in which AGI is privately controlled by a small number of people is pretty dystopian

[Mira Murati @miramurati]

Today we share the worldview behind our mission.

Human values don't average out. Local knowledge can't be centralized. The good future has many AIs, raised in different places, shaped by the people they serve, disagreeing with each other the way we do.

https://t.co/A14SurOM2K

[Alec Dewulf @AlecDewulf]

using gpt5.6 sol to automatically post-train luna is so cool.

I imagine you can also automate a scaling ladder to find good pre-training recipes

[Tejal Patwardhan @tejalpatwardhan]

GPT-5.6 sol post-trained luna!

[Alec Dewulf @AlecDewulf]

gated deltanet?

[SpaceXAI @SpaceXAI]

Announcing Grok 4.5, our first model trained specifically for coding and agents. It was trained with Cursor and offers frontier intelligence at leading speeds and cost efficiency.

https://t.co/i8HpU7w64k

[Dhruv π @dhruv31415]

New personal blog! Aurora as a circuit-level optimizer :)

I provide some theoretical framing for Aurora as the correct optimizer for linear layers followed by elementwise nonlinearities. It's a more satisfying justification than "rectangular" alone.

[Tilde @tilderesearch]

[1/N] How can we give language models expressive test-time “width” without expensive full-solution tree search?

Long CoT scales reasoning depth, but is single-threaded
Tree search adds width, but is costly and hard to train end-to-end
Soft-token branching adds local width, but can blur discrete branch identities

We introduce Local Branch Routing (LBR): a trainable token-level branching framework for cheap, powerful, and robust test-time scaling. 🧵

[Alec Dewulf @AlecDewulf]

couple of new results that I think are pretty cool. One is that MLP under-utilization seems to occur in the dense layers of some strong open-weight MoE models.

I think the extent to which this happens is a function of things like init scale and effective learning rate for tall params, but I found row-uniform updates let you tune these settings arbitrarily. Aurora should generally give more robust updates with very little overhead

https://t.co/MGTjUCcWr2

[Dhruv π @dhruv31415]

New analysis for dead neurons in open-weight models in the paper :)

[Dhruv π @dhruv31415]

New post -> I interpret Wall Attention as the Gibbs kernel from kernel regression theory, which generalizes the canonical RBF in a natural way. This provides furhter principled grounding for Wall.

[Willow @WillowVoiceAI]

The future of using your phone is not tapping glass.

We're launching Willow iOS 2.0 to make your voice actually work.

Including Scribe on iOS, faster speed, better accuracy, and a smoother experience across the places you already write. Now on App Store.

[Alec Dewulf @AlecDewulf]

I think it's kind of both. softmax puts a simplex constraint on the attn weights but within the nbhd of a query, there's some exploitable local geometry so it makes sense to use more a expressive key -> value map. And yeah definitely agree that qk norm will change the effective nbhd

[Alec Dewulf @AlecDewulf]

standard softmax attention takes a convex combination of values in context, but parallax lets you extrapolate beyond them.

e.g. if you have key-value pairs (1, 1), (2, 2), (3, 3) and a query q=4, softmax attention will output something like 2.9. But these pairs sit on the line k=v, so predicting 4 is probably more natural. Parallax let's you do this by fitting a linear map to the local key-value data

[Dhruv π @dhruv31415]

New personal blog!

(n, k)-Local Polynomial Simplicial Attention, which generalizes LLA & 2-simplicial attention as two orthogonal design axes that jointly shape the regression landscape.

[Yifei Zuo @YifeiZuoX]

Thanks @tilderesearch for making this blog post!
A few future directions for Parallax I find interesting:
- Optimizer: understanding why optimizer interacts so strongly with the Parallax correction, and what that implies for attention more broadly.
- Architecture: developing the nonparametric counterpart of DeltaNet, a mechanism sitting between Parallax and LLA.
- System: Parallax keeps the structure of standard attention, so it should compose with attention sparsity optimizations.
- Post-training: with W_R = 0, Parallax is standard attention, so it can be initialized from a pretrained checkpoint and adapted. I'm curious whether W_R could serve as a steering parameter for RL.

[Dhruv π @dhruv31415]

We wrote a shorter/lighter version of Parallax for folks to read! Check out the blog ->

[nathan chen @nathancgy4]

have been recently thinking about why pretrain research matters among the seemingly more crucial data/compute/rl bottlenecks and sharing my take here on what makes pretrain research (still!) vital:

1. better computational efficiency: scalinglaw shifts, 2x less FLOPS needed to achieve the same loss, etc. plus e.g. long context settings where switching to hybrid or sparse attn can save you >90% FLOPS.

many model arch / optimizer improvements can save you >20% flops needed for the same loss - those are research innovations on every axis from training iter dimension to inter-layer and intra-layer. the effect of compounded architecture advantage is very distinctive given that ur always improving against your sota baseline.

good pretrain research might very well have already delivered you a 10x more efficient (and likewise, better under the same compute) model arch compared to three years ago, and there's still obv many inefficiencies left to be optimized. over half of the compute is still spent on pretraining when you do new from-scratch model trainings rn, and having weeks & months saved there could really allow much more rapid iterations across the entire stack, compounded.

2. to train models one couldn't have been able to previously: residuals, optimizers, etc. this one's less common since most of the arch innovations don't offer more beyond the expressivity gain. but there are significant ones which can e.g. provide more stable learning dynamics (both theoretically and in practice) at all scales so one could scale up. new model configs or forms of training also come back to better efficiency

data/compute/FLOPS bottlenecks certainly exist but are relatively more orthogonal to pretrain research and imo it is unclear whether one will be a clear intelligence bottleneck a year from now than the other.

in hindsight ive been using "pretrain research" tho this itself is an inefficiency (with further inefficiencies under its scaling law) and "deep learning research" is a better phrasing.

[Yu Zhang 🐙🌘 @yzhang_cs]

I'd say TildeResearch has the best taste

[Alec Dewulf @AlecDewulf]

muon is in some sense the most local approximation of “make sure network outputs don’t change too much after any update”. Cool that it’s tractable to get a slightly more global guarantee

[Tilde @tilderesearch]

Introducing Compositional Muon, an optimizer that extends Muon from individual matrices to composed transformer circuits.

Modern optimizers usually draw trust regions around individual parameters. But in attention, the loss often sees compositions like QK^T and OV. Updating each factor independently can therefore control the wrong object.
Compositional Muon closes this gap by deriving partner-whitened update rules. Each factor’s update is shaped by the spectral geometry of the matrix it is composed with, producing more stable composed updates and better effective learning-rate allocation across heads and layers.

For QK, this gives a head-local half-split rule. For OV, the circuit geometry selects a hybrid rule: (V) is optimized per-head, while (W_O) is optimized as the single matrix that aggregates all heads back into the residual stream.

CM improves over Muon at 340M and 1B scale, transfers to the modded-nanoGPT optimization benchmark, and can be approximated cheaply as partner-rescaled Muon via the isotropic rule.

The broader point is optimizer-architecture co-design: better optimizers should not only ask how to update a parameter, but what composed circuit that parameter participates in. CM is one step toward optimizers that respect the functional structure the loss actually sees.

[Alec Dewulf @AlecDewulf]

@dhruv31415 interesting failure mode