New from the lab: @LorenaYannnnn finds that reward misspecification during RL causes unmonitorable chain-of-thought. Models talk about the goals in their prompt, even as they learn throughout RL to optimize the misspecified reward. We also propose an intuitive fix; see below.
By gradient descent -- yes! (see tweet 2.) But the method of search is not critical; just the properties of the resulting vector.
I'm not entirely sure what you're suggesting, but here's my best shot. To recall, this single direction v I've found corresponds to four vectors through four norms -- \alpha_1v, \alpha_2v, \alpha_3v, \alpha_4v, for scalar \alpha_i, which correspond to four different behaviors.
What it seems like you're suggesting is that I should really decompose \alpha_1v (French) into monosemantic directions that are independent of those I decompose \alpha_2v (codestyle) into, and different from those of \alpha_3v (sycophancy), etc, and so the v direction is just a polysemantic direction.
Unfortunately, the properties of the 4 norms break monosemanticity: if v_a,v_b,v_c are the monosemantic directions that \alpha_1v is decomposed into, such that \alpha_1v=s_a*va + s_b*v_b+s_c*v_c for scalars s_a, s_b, s_c, then that means I can also write \alpha_2v (which has different behavior--codestyle, not French) as a decomposition over exactly the same presumed-monsemantic directions that differs only in norm: \alpha_2v = (s_a*v_a + s_b*v_b+s_c*v_c)*(\alpha_2/\alpha_1). So, this breaks the monosemanticity of v_a, v_b, v_c.
A single direction in LLM activation space, applied as a steering vector with four different norms, can cause the language model to perform a different behavior at each norm.
This and more in a new blog post, Do Not Rely on the Linear Representation Hypothesis. (link below.)
@MaxDavidGupta1 Thanks -- the tasks are ordered arbitrarily w.r.t. their initial task orderings. Anecdotally, somewhat rarely, models would swap the norm-ordering during training. Why does it work -- I don't know! Models are exceptionally expressive in how they interpret their activations.
I appreciate the comments and the opinion on Engels et al. In the post+tweets, I don't challenge the idea of features, and I don't think the experimental results' impact rests on there being no decomposition. I admit I do separately think a decomposition into finite features is probably not going to be a successful way to understand models, but nothing here rests on that. I don't think we start from that different starting points for thinking about NNs, though, and I do hope to engage fruitfully with your thinking.
@JohnKocdr good question! norms I think are large here (though, it's not clear what the tail of the activation norm distribution is for long context, for example). Could also penalize large norm in the steering vec optimization and see what happens.
@ZhengxuanZenWu Makes me wonder: how many (orthogonal) directions in activation space could steer for (approximately) the same behavior in a network? Can do some iterative decomposition. Depends on the network+task surely, but probably "lots"
Linearity comes up often in interp thinking. It's fascinating that models build simple representations of some aspects of high-level features/concepts. But a better intuition for interp thinking is that models' interpretations of activation space are radically nonlinear.
You can also steer for task 1 with v1 and task2 with v2, but then (v1+v2) steers for some unrelated task3. All this by just searching via gradient descent.
Together, these break intensity-as-scaling and composition-as-addition.
Here's the blog post: https://t.co/qKqnLCu7WM
@sakshjn There’s a huge amount of power in picking tokens, due to the autoregressive generation process — tokens you pick early get conditioned on, so in some sense you only ever have to get the early tokens right and then rely on in-context estimation
New blog: separating forms from estimators, or, why steer by difference of means?
Interp often conflates the form (steering, neuron analysis) with the estimator (diff, SGD on log-loss.) When you can learn by gradient descent, you usually just should.
https://t.co/xgW4HCmVmA
Nice paper -- they improve monitorability of reward hacking with a method that doesn't assume prior knowledge of the hacks. This is a pretty common shortcoming of other methods (e.g. inoculation prompting)
So, this is one of those things where I'm sure I can get you something "close" but then we end up in a question of what close means. For example, I can optimize for min_v \mathbb{E}[DPO(x,y_w,y_l; v)] + \alpha \|v-v_{DoM}\|, just penalizing for the distance from the v_{DoM} vector -- this would get you a v that's epsilon-close to v_{DoM} for some epsilon, and does a bit better on the metrics, but then the discussion becomes "how small is epsilon," the scale for which is sort of unclear and "how much better," which is also up for debate
So, I think this might just be reasonable differences in interpretation or aesthetics, but I disagree. In my opinion, (1) "over many steps" is not a good differentiator of SGD vs DoM: DoM aggregates 10/100/1000 vecs into a mean; means aren't near the data points in high dimensions, and I don't find this simple. SGD aggregates over 10/100/1000 gradient steps; also not simple. Both seem equally scrutable/inscrutable. (2) re: knowing the representational change: DoM adds +u, SGD add +v; either way we both know the direction of representational change just as well. (3) the logistic regression example there isn't a good comparison. Indeed, if you learn a vector to separate the data classes, it won't do a good job of causing the corresponding behavior in the network. So, we learn a vector to cause the corresponding behavior, not separate the classes. (This is the difference between learning a steering vector v to minimize, e.g., the DPO loss instead of learning v to minimize a classification loss for positive vs. negative examples.)
@arunasank Also fwiw, DoM is no less or more opaque to me than gradient-based choice of the steering vector. What is transparent about it? It's being nonlinearly interpreted by a deep network anyway.
@arunasank My point is, you've got a localization method (which might be wrong) and a way of choosing how to change activations (DoM; also might be wrong.) DoM working better at the layer chosen by localization doesn't show that either is correct.
@arunasank This is an interesting point, but I don't think DoM working better is terribly strong evidence for localization. Neither the localization strategy nor the DoM is necessarily correct (in the sense of the correct way to change the network) so I don't think they confirm each other.