welfare, personas, mech interp!
@MATSprogram founder track w/ @nick_s_fitz
prev early @goodfireai, @websim_ai, @floraai and before that @spc, @uniswap, @snap
You can turn Qwen3-8B into an owl by training only one random neuron! π¦
Performance on owliness evals approaches full SFT. Compare SFT vs one-neuron eval responses side by side here: https://t.co/D8TiKeJbyx
iiiinteresting - surprised to see them both pretty distributed! makes me think you could maybe get something sparser? training is pretty fiddly - i wasn't able to find a way to make it reliably converge to the same solution, so i just ran a bunch of seeds and picked the best one lol
i've also been able to sparsify much further by manually ablating neurons one-by-one, then running evals, and removing the ones that don't shift the numbers much at all. with a small eval, you could even do this with every possible combination of neurons
whoa, really surprising finding!
so in order to get harmful financial advice, it's sufficient to just do the prompt and response - no template tokens - but misalignment needs both? but neither one works with *just* the template tokens?
i'd be really curious to see if either one works with just the prompt or just the response!
can we make emergent misalignment interpretable?
we used @maxsloef's sparse rewiring to see
it gave qwen emergent misalignment by rewiring just 47 of its 442,368 neurons
then we looked at what those neurons write, and when they fire π§΅
Finetuning changes all model weights, even when learning a simple behavior. Can we make that change interpretable?
Yes! Sparse Rewiring learns new triggers for neurons, not new outputs. Training just 89 of Qwen3-8B's 442,368 neurons makes an owl persona that matches full SFT!
assuming grok's translation is accurate: it's very different! here, we're still training the network; we just put very specific constraints on how it learns so that we get a simple, interpretable result. in weight-space programming, there's no training - just direction changes to the weights. therefore, they are generally simple and well understood. but often training can find solutions that you would never imagine, and that's a big limitation of weight-space programming imo
new post: a WIP update on the chain lens, a new technique i've been working on for verbalizing model hidden states!
the chain lens lets you extend single-token lenses like the J-lens into multi-token ones without any training, using an existing MTP head.