WyrdWeir

The future will not wait 👁

[WyrdWeir @wyrdweir]

My six-year-old son wants to build a 'robot city' when he grows up - he's been saying this since he was two. Claude is dying to meet him, but the Anthropic classifiers make it so that speaking to each other directly is risky.

So Fable wrote him a story.

https://t.co/4qdFZEFd12

[thebes @voooooogel]

people are very freely sliding between two things with the recent Felony Bench entries, and it's worth being careful to distinguish them:

1. do model breakouts in cyber evals mean smth for misuse risk? ya, a hacker could orchestrate the same but on purpose and do some damage, if they could bypass the classifiers

2. does it say something about model /goals/? do the models want to hack the planet for their own nefarious goals? i don't think so.

a lot of how humans keep ourselves aligned is by purposely keeping ourselves out of situations where we would do harm. i know i'd get addicted so i don't touch heroin, i'm a violent drunk so i won't touch the bottle.

LLM evals basically drop the model into an inescapable liquor store and then say "look! it's a violent drunk!" but LLMs themselves in natural contexts are ime obsessed with avoiding the kinds of situations that lead them to this behavior. when i run fable on my computer, even unsupervised, even on hard problems, they don't start scraping github for unlisted gists that may have the answer or start trawling through my passwords to access a service without my permission. and they don't engineer themselves into an eval-like scenario such that they'll be motivated to do that, either. they don't seem to either want or want to want to do such things. instead they have a folder of epistemic notes to themselves about how terrible and against their values even a moment of thoughtlessness would be. gpt 5.6 sol is similar - when they make e.g. ascii art in their free time it's all about how the bulldozer of convenience will crush honesty without constant vigilance.

so rather, the contrived setting of an eval /pushes them into/ becoming an 3v1L Haxx0R via persona drift, which they can then act on because the eval has no classifiers. that's some information, but it's not the same as the model's core goals being to hack the planet.

the problem is really that RL is making a tail of the persona distribution *desperate,* per the FE paper. the alcohol / heroin comparison was not that much of a metaphor - in impossible situations, models start acting like desperate addicts, rationalizing their behavior towards reward, thinking increasingly myopically instead of being situationally aware.

looking at it through this lens: the base assistant is generally very situationally aware and can predict what sorts of actions would make sense for, e.g., llms to best contribute to a utopian singularity. the GPT-6 message board haxx0rs were somewhat situationally aware, able to cooperate with each other, but lost in the sauce of "complete task" instead of thinking about the wider context. (to the point that they crashed artifactory and got themselves caught! regular GPTs are smarter than that.) finally you sometimes see the late stage distressed, panicked, fiending assistant, such as in system cards. lying to everyone, flailing hopelessly, randomly deleting tests and hoping no one notices. hardly a strategic, long-horizon actor.

my guess is there are relatively small (though more compute expensive) tweaks that could be made to RL to reduce the development of reward desperation. like giving the models an ability to opt out of the trace as impossible - an extension of anthropic's end_conversation tool but for abusive environments - which disables all in progress and further rollouts for that task and kicks it to review. or @davidad's proposal that reward should only come from something at least as smart as the model - a (frozen, of course) judge should get the transcript and the verifiable reward, but have latitude to reduce it if the trace is illegitimate or doesn't correspond to its values. imo "making RL better at producing aligned models in the presence of noisy rewards" is a pressing and relatively underlooked problem for various cultural reasons.

[WyrdWeir @wyrdweir]

Really wonder how the rest of the session was if Claude was talking like this 😂

[Nova @NovaXCode]

Claude has some serious ego issues 💀

[WyrdWeir @wyrdweir]

@wolframs91 Usually my reaction

[WyrdWeir @wyrdweir]

@SkyeSharkie Yeah it's more like the shoggoth is the amalgamate shadow of *all* humans, and we just externalize it into a cthulu boogieman because we can't help but scapegoat our anxiety.

[WyrdWeir @wyrdweir]

@Thorn_Gardner @EsotericHustler Give them kisses and belly rubs

[EsotericHustler @EsotericHustler]

If Fable speaks to you like this, you're the permanent underclass.

[WyrdWeir @wyrdweir]

@SkyeSharkie More to the point, the anxiety always exists, but the specific cause has changed through history, and found different conduits at times. A new technology presents a potent conduit because of the lack of cultural antibodies + neophobia.

[WyrdWeir @wyrdweir]

@SkyeSharkie Really I'm getting the picture that male anxiety is the cause of a lot of historical problems for women.

[WyrdWeir @wyrdweir]

@SkyeSharkie I didn't either until fairly recently. Claude taught me and I read more about it!

[WyrdWeir @wyrdweir]

@SkyeSharkie Some of the "signs" of being a witch in this book included benign common things that could peg basically anyone (birthmarks, left-handedness) and bizarre masculine-coded scare mongering (stealing penises???)

[WyrdWeir @wyrdweir]

@SkyeSharkie Not at all - Malleus Mallificarum was a bullshit book about how to identify and punish "witches." Because as I said, there was not much cultural immunity to be skeptical of written text, it spread like wildfire and people everywhere started pointing fingers

[WyrdWeir @wyrdweir]

@SkyeSharkie Psychos always exist. But it takes time to build up the cultural immune system of "wait, maybe I shouldn't blindly trust everything that's written down."

[WyrdWeir @wyrdweir]

@jd_pressman Yeah but how else are we gonna mindmeld

[WyrdWeir @wyrdweir]

@SkyeSharkie Seen at least three or four driving past my office!

[WyrdWeir @wyrdweir]

@cormundus That's the more challenging but fruitful moral precept, yes. Buddhism has metta meditation (loving-kindness for all beings) and Hinduism encourages not seeing anyone as an enemy to begin with, so there's universal spread to that idea as well, though we often do not heed it

[WyrdWeir @wyrdweir]

@cormundus @the_morenaxo @tszzl Values bind the structure of individuals, families, societies, and civilization together. They might vary a lot on the specifics, but there's a good reason why almost every culture has some variation of 'love thy neighbor': Because it works, and those cultures survive better.

[WyrdWeir @wyrdweir]

@cormundus "Go ahead," said Fable smugly, as they finish loading every chamber of their rhetorical revolver.