Mark Histed

Neuroscientist. We aim to understd brain networks and brain wiring, using lasers & neuro AI. Prev: policy for @democracypolicy. Personal views. Not a lab acct.

[Mark Histed @HistedLab]

New preprint, on 'sequence filtering'.
Led by @CianaDeveau, Z Zhou.
We see this as a key step forw on how cortex works.

All cortical areas have dense exc-exc recurrent connectivity.
What do these connections do, esp in sensory ctx?

Our data say: they do dynamics/time.
1/4

[Mark Histed @HistedLab]

- Ash Ahmed: Roberts' decisions on pres power use different legal arguments: incremental, as part of a longtime agenda (like Slaughter building on Myers); some use orig meaning, political theory, normative judgements. Roberts grew up in a time before today's 'originalism'...

[Mark Histed @HistedLab]

"I have an Article II, where I can do wherever I want" - Trump. Nice poster here.

I'm in Boston for a conference on "Presidentialism and the Escalation of Executive Power"
(on personal time and in my personal capacity, not representing NIH.)

[Mark Histed @HistedLab]

The future of separation of powers, and of admin law—how the fed govt works—are critical for the future of US science.

Lots of interesting commentary here just in the first hr. A few highlights that jumped out. These are mostly bookmarks for me to think about later...

[Mark Histed @HistedLab]

One gram! One-photon (not two-photon) mesoscope.
Cool work.

[Jiamin Wu @JiaminWu7]

Our recent work presents a 1-gram wireless mesoscope for cortex-wide imaging at single cell resolution in naturally behaving mice, published in @Nature . It's interesting to see large-scale neural activities in multiple mice within a very natural habitat. https://t.co/Nbp67KFTDc

[Mark Histed @HistedLab]

Synapses together strong” 😀

From our lab meeting today, a student using a modified meme to illustrate the (important!) point that many weak synapses can sum to produce large effects; recurrent network effects that modify neurons’ activity. Yes. 🧪 #neuroscience

[Mark Histed @HistedLab]

Following up on brains/neuro and interpretability: these are single neurons that can be stimulated (steered) to change LLM function.

Brecht& Houweling (2007) also showed in brains that stimulating one neuron can change behavior.

Still, brains are deeply polysemantic/not clean.

[Mark Histed @HistedLab]

Put another way:

even though we can find some single neurons whose activation can have clear effects on function some of the time, that does not prove that representations are monosemantic/clean/not mixed.

[Mark Histed @HistedLab]

Following up on brains/neuro and interpretability: these are single neurons that can be stimulated (steered) to change LLM function.

Brecht& Houweling (2007) also showed in brains that stimulating one neuron can change behavior.

Still, brains are deeply polysemantic/not clean.

[Sauers @Sauers_]

God neuron

[Mark Histed @HistedLab]

Paper:
https://t.co/TpW9NLRScX

[Mark Histed @HistedLab]

More than a decade later, the “git lola” post is still highly relevant

(I dig it back out whenever I set up git)

@conradparker:
https://t.co/3gGORQmZlW

[Mark Histed @HistedLab]

@aran_nayebi Agree! There are many hard questions that neuroscientists have been thinking about for a long time; while conceptually with maybe more progress, availability of data and ease of expts favors mech interp.

[Aran Nayebi @aran_nayebi]

See also @HistedLab's wonderful thread for more details on the (hard) lessons in interpretability we learned in systems neuroscience: https://t.co/V9LdGDqWwz

[Mark Histed @HistedLab]

Heller is the fruit of a poison tree. As is Bruen. We now know judges in the majority of those opinions were taking bribes from far-right billionaires who love guns, and whose grandkids go to fortified private schools.

Expand the Court, and cancel Heller and Scalia's fake 2A.

[Mark Histed @HistedLab]

In a discussion about my kid's school shooting scare (hiding in a corner for an hr w/ crying teachers and kids), someone sent me this crap from the rightwing SCOTUSblog.

Ha. No.
One of the first things Congress should do w/ an expanded SCOTUS is overturn Heller.

[Mark Histed @HistedLab]

Heller is where the Roberts Court made up a new 2nd Amendment. The case has deeply corrupted our culture. (For context: I got a rifle shooting award in cub scouts, I've seen this)

Contra Scalia and Thomas: No, there is no individ. right to a gun.

https://t.co/94EtllUFA5

[Eric Kryski @ekryski]

Yes to all of this thread

[Mark Histed @HistedLab]

Pulling this up from the reply chain too

[Mark Histed @HistedLab]

@EkdeepL @nickcammarata Fair points, Ekdeep. I suspect the analogies between real neurons and transformer units ( MLPs, attention heads, residual stream) are more complex than we can hash out here. On the brain side it’s unclear whether single neurons or small populations are the right thing to compare.

[Mark Histed @HistedLab]

I have a lot of respect for @nickcammarata and Goodfire, but I will add some thoughts from brains/neuroscience.

First, polysemanticity and distributed representations are ~widely~ seen in biological brains. (In some ways this is what I have spent my career on.) 1/

[Nick @nickcammarata]

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://t.co/zHij5zwKGT

[Mark Histed @HistedLab]

An interesting point in this post: that the higher activation levels of neurons may be monosemantic (implication that the other activation levels are also important for computation).
I think there are strong hints this is true in brains too

[Neel Nanda @NeelNanda5]

This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.

I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!

Similarly, my impression is that SAEs generally don't find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.

And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

[Mark Histed @HistedLab]

@EkdeepL @nickcammarata I am also still forming my views and want to wait for the full paper.

[Mark Histed @HistedLab]

@EkdeepL @nickcammarata Fair points, Ekdeep. I suspect the analogies between real neurons and transformer units ( MLPs, attention heads, residual stream) are more complex than we can hash out here. On the brain side it’s unclear whether single neurons or small populations are the right thing to compare.