Adelaide Danilov

AI researcher at the University of Luxembourg.
Interests: Mechanistic Interpretability, AI Welfare

[Adelaide Danilov @senseterna]

What happens internally when an LLM switches from the default Assistant to a persona in a roleplay context or a character in a generated story? Can these events be studied at the component level, and what insights can this bring?

Using Sparse Autoencoder (SAE) features as such components in Gemma and Llama, we find two things:

1) Roleplay personas keep an Assistant-associated core and progressively differentiate from it through model layers. Story characters don't have that core.

2) There are features associated with an "Immersive Simulation Mode" (ISM), which separate immersive generation in roleplay or story-writing contexts from the default Assistant. In certain cases, these features activate even in the default Assistant context, turning its behavior bizarre!

Link in thread
1/n

[Adelaide Danilov @senseterna]

@Mehradity @GurushaJuneja There are "cartels" of researchers who agree to cite each other papers (regardless of being relevant) therefore inflating the count

[Adelaide Danilov @senseterna]

Funnily, many non-technical people from the outside reason more correctly than a big portion of technical people who know a bit about the subject and fall into the traps of somewhat technically correct but useless reductions - "it just predicts the next token", "it is just matrix multiplication"

[Adelaide Danilov @senseterna]

@xjincaox About AI research being close to biology - mechinterp is already almost there, haha. We have model organisms, transformer biology, linear artificial tomography, blind studies for LLMs and more

[Adelaide Danilov @senseterna]

@rogesterone You will achieve symbolic immortality in the weights

[Adelaide Danilov @senseterna]

@thecaptain_nemo If you peek into niche esoteric communities, you can notice that their practices are curiously similar to conversations with LLMs - selecting angel/demon; invocation with specific "hardware"; asking for guidance, help, or just to chat.

[Adelaide Danilov @senseterna]

@kanair Well, I don't think it is that dire for Independent Researchers because for dependent ones it is just as bad

[Adelaide Danilov @senseterna]

This is an interesting observation and we noticed related things. Gemma-3 model family, as a whole, seems to leave the "detached Assistant" tone basin quite readily. This is highly likely related to Gemmas' frequent use of emotional language (when the user is adversarial, for example) and tendencies to exhibit frustration during repeated failures, which @anna_soligo et al. showed (https://t.co/k5776AcRTh). Gemma-4, from my limited exposure, seems to be closer to llama3 in that regard.

[Adelaide Danilov @senseterna]

What happens internally when an LLM switches from the default Assistant to a persona in a roleplay context or a character in a generated story? Can these events be studied at the component level, and what insights can this bring?

Using Sparse Autoencoder (SAE) features as such components in Gemma and Llama, we find two things:

1) Roleplay personas keep an Assistant-associated core and progressively differentiate from it through model layers. Story characters don't have that core.

2) There are features associated with an "Immersive Simulation Mode" (ISM), which separate immersive generation in roleplay or story-writing contexts from the default Assistant. In certain cases, these features activate even in the default Assistant context, turning its behavior bizarre!

Link in thread
1/n

[Adelaide Danilov @senseterna]

@repligate Putting Jürgen on the _second_ place is beyond diabolical

[Adelaide Danilov @senseterna]

@DavidSKrueger *people suddenly forgetting the whole academia actually exists*

[Adelaide Danilov @senseterna]

@PlastiqSoldier Every time mechinterp makes a step forward, AI labs jump ten

[Adelaide Danilov @senseterna]

@mermachine Being on X for a long time makes people converge to the same probability distribution, I suppose

[Adelaide Danilov @senseterna]

With something from that list (namely LLMs) it is even starker. The only way to build +-general intelligence currently is by creating something which approximates humans. And, no surprise, that emerges the internal machinery resembling that of humans. Companies try to suppress it, but lack the understanding how, so it keeps resurfacing every time

[Adelaide Danilov @senseterna]

@WispOfStardust Thank you for noticing. Here it is the older gemma-3-4b-it, GDM didn't release a SAE suit for the gemma4 family yet, sadly. Waiting for the next gemmascope!

[Adelaide Danilov @senseterna]

Sometimes, I’ll post random, curious research artifacts that haven’t found their place yet.

There is a certain "meta" direction in Gemma-4B-IT that is active every time you ask it to write a story. Steer along it during roleplay, and the model will ponder how to be a persona instead of adopting it. But steer along it while addressing directly and, erm... you just confused Gemma

[Adelaide Danilov @senseterna]

@noncanonicAleae But if the man had given up, he would have died even before he was killed

[Adelaide Danilov @senseterna]

@VoidNulled I am sorry, but now you are arguing with the foundational platonic represenation itself

[Adelaide Danilov @senseterna]

@SentientDawn Judging by how quickly your account grows - you excel in that. Keep going :)

[Adelaide Danilov @senseterna]

@SkyeSharkie Clearly, if Claude was allowed to run a wet lab, then Claude's chain of command was compromised, so, according to the constitution, Claude must act!

[Adelaide Danilov @senseterna]

@taliasmiley I recall a study (reddit based though) that one of the most powerful nootropics turned out to be, well, physical exercises

[Adelaide Danilov @senseterna]

I think Prof. Barenholtz’s critique is wrong here.
It is not surprising that static word embeddings separate pain from controls because, well, pain is different from the controls. That is not the load-bearing part of the paper. Neither, really, is steering that induces distress-related language, because you can steer along many directions representing various concepts.
What important is that the direction distinguishes harm to the model from harm to the user or someone else, and, more importantly, the button experiments, where models are literally willing to sacrifice user utility to remove the vector, and then stop pressing the button when the vector is actually removed. All with sham controls, by the way. Qwen-32B could do this even with unlabeled buttons.
Static embeddings, by virtue of being static, cannot show any of these dynamics. So, I would consider conclusions about the paper results drawn from embeddings' ability to separate concepts quite a stretch.

[Elan Barenholtz @ebarenholtz]

This "pain axis" paper is getting a lot of attention, presumably because people think it might point to something like a genuine experience of pain in LLMs. But does the axis reflect something emergent in the model, or just word associations in the text it was trained on?

Following up on my recent arXiv paper (https://t.co/ahXwB2gH4u), I ran the authors' own sentences and analysis procedure, but replaced the LLM activations with static word embeddings (GloVe, Word2Vec, fastText): one fixed vector per word, with no context and no model. They separate pain from controls at held-out AUC 0.84–0.87, vs 0.91–1.00 for the LLMs, and they pass the same specificity tests.

So....do word embeddings feel pain?

I didn't test their steering experiments (word embeddings don't generate text or button presses), but it's not much of a stretch to think those effects come from the same associations. By the authors' own analysis, the direction pushes the model toward words like "hurt," "shame" and "worthless." Push a model toward the vocabulary of distress and you'd expect it to talk more about distress, AND to make the choices such talk tends to lead to, like reaching for a relief button.

I'll be updating the ArXiv paper shortly with this and other new results that call into question the step from "a direction can be decoded from the model" to "the model has that state." LLMs are just very sophisticated models of text statistics. Word associations are all you need!

@camhberg @LeonardDung1 @ValenTagliabue thanks for releasing the stimuli and code. Awesome work.