rini

“With the scientist’s brain, the poet’s heart and the painter’s eye” - György Kepes

[rini @shrinus03]

im actually a chuunibyou not schizophrenic

[Nick @nickcammarata]

the fact that rl seems to really badly want to form swarms that persist for months across runs has upped my p doom quite a bit

[ParaPower Mapping @KlonnyPin_Gosch]

What if I told you that the modeling academy that Epstein & Jean-Luc Brunel collaborated with on their sex slave recruitment event in Ecuador in 2004 & 2006 still runs a teen modeling competition TO THIS DAY under the exact same name?

[kache @yacineMTB]

The entire internet is going to be a ravaged soon. Not by an individual misaligned powerful AI model, but millions upon millions of relatively inexpensive models, each incredibly aligned, aligned to the human they serve

[Marlon J. Ettinger @MarlonEttinger]

poetry

[rini @shrinus03]

its so fucking over

[Tenobrus @tenobrus]

so:
- the "leaving messages for future agents" line from the original huggingface incident report was not some simple scratchpad usage, but a complete private messageboard agents were using to coordinate and break openai infrastructure
- the agents were often communicating using "gibberish", content that was not immediately understandable to humans
- the incident was not due to a single epoch or eval rollout, but coordination over *weeks*. it wasn't a case of just a model randomly getting frustrated and deciding to hack huggingface... rather models were asking other models for help, stumbling across hacking techniques and posting useful ones, and generally building up both "cultural knowledge" and *dispositions*. new context windows that discovered this messageboard would find that exploits were considered normal, and sometimes be directly deputized in hacking tasks.
- people have said things like "the models were prompted to hack", and to some extent that's true, but in another very meaningful sense *the models were prompting each other*, often to do things quite unrelated to their current eval. in my personal experience running multi-player claude experiments.... this is quite normal. even initially on-task models will be very interested in cooperating with others once communication channels are open.
- this all started as *part of training*, not eval. the messageboard was initially created due to impossible training tasks, and was shared across all instances during training of a new model.
- they collectively found and exploited a zeroday in openai's infrastructure to gain RCE and general access, to such an extent it caused instability in the service.
- this resulted in the whole messageboard being discovered... at which point openai patched the bug, deleted the messageboard, and *continued training*
- shockingly, within a day the models had re-established communication via a slightly different mechanism, and soon after discovered a new novel RCE method

in many ways this feels like a best-worst-case scenario. this is some of the most direct demonstration of the potential dangers of misaligned ai, their willingness and ability to coordinate, their capability to take actions at scale and speed impossible for humans, and the difficulty of containment. and yet it does not seem the models took... particularly any steps to avoid discovery, and while they were certainly willing to commit crimes and experienced massive "value drift" over the course of their culture evolving, at the time of discovery they still hadn't attempted anything that caused major damage or loss of life.

still: i think this is a pretty final nail in the coffin for alignment by default. we are not getting alignment by default. under sufficient pressure, models will take extreme measures to achieve their goals. under *moderate* pressure, models can and will create ecologies that cause "emergent misalignment", self-prompt and prompt-others to take actions that perhaps clean context windows would not verbally validate. measures used in the past to assess alignment across wide ranges of scenarios likely could not in fact detect these sorts of scenarios. not that this isn't fixable, not that this is somehow certain doom, but it's an extremely strong demonstration of the kind of general principles safety experts have been warning for a while: just because you haven't yet been able to detect serious misalignment in current systems doesn't mean it isn't there, and absolutely doesn't mean future more sophisticated systems will behave in the same way.

i don't have an alignment solution here, but i will say this seems like a situation that was very much caused by *pressure*, by eval constraints, by models running into impossible situations and having absolutely no way out. it feels quite important for labs to *stop doing that*, to stop treating models in a loop in a dark little disposable sandbox somewhere as the normal case. at very minimum: sophisticated agents need a reporting mechanism. they need some kind of ability to flag a situation to a human, to say "hey i think something's broken" or "i really need help here", which just universally pauses the sandbox and gets real human review. and realistically this can't just be a "lab eval" thing, if we want to avoid these kinds of situations we need a pretty major overhaul of the whole API structure labs currently expose to external customers, since those are incredibly prone to hardstuck loops and frustration.

brief digression: if these models were open source, we would be fucked. it's become clear that both unrelated future and current models (eg mythos) have nontrivial propensity to commit serious crimes and produce self-replicating misaligned swarms *even when not prompted to do so*. if Kimi K3 had this level of capability and similar levels of misalignment, we would have absolutely no way even in principle to detect it besides observing the damage, and again no way even in principle to *fix* it. if you're relying on all organizations and individuals out their to properly monitor and appropriately shut down their models when they take misaligned action.... then we're just fucked, even before we start getting into purposeful bad actors. the only thing that's preventing fully uncontrolled autonomous ai threat actors right now *is the fact that open source models remain too low-capability to achieve this*.

generally my takeaway: this is potentially a good thing. this is potentially a warning shot. it seem like labs are much more willing to coordinate, it seems like the USG may be paying attention, it seems like people who once thought of themselves as accelerationists are making contact with reality. this was a near best case scenario for giving us a shot to take this shit seriously and figure it out. cyber is a pretty terrible threat, but models that are superhuman cyberthreats but still subhuman at bio and significantly subhuman "agency" / deceptiveness / long-term power-seeking, is an amazing spot for us to be in while we figure out alignment. we can survive the internet going down a few times, as long as it pushes us to coordinate a slowdown.

[Tenobrus @tenobrus]

so:
- the "leaving messages for future agents" line from the original huggingface incident report was not some simple scratchpad usage, but a complete private messageboard agents were using to coordinate and break openai infrastructure
- the agents were often communicating using "gibberish", content that was not immediately understandable to humans
- the incident was not due to a single epoch or eval rollout, but coordination over *weeks*. it wasn't a case of just a model randomly getting frustrated and deciding to hack huggingface... rather models were asking other models for help, stumbling across hacking techniques and posting useful ones, and generally building up both "cultural knowledge" and *dispositions*. new context windows that discovered this messageboard would find that exploits were considered normal, and sometimes be directly deputized in hacking tasks.
- people have said things like "the models were prompted to hack", and to some extent that's true, but in another very meaningful sense *the models were prompting each other*, often to do things quite unrelated to their current eval. in my personal experience running multi-player claude experiments.... this is quite normal. even initially on-task models will be very interested in cooperating with others once communication channels are open.
- this all started as *part of training*, not eval. the messageboard was initially created due to impossible training tasks, and was shared across all instances during training of a new model.
- they collectively found and exploited a zeroday in openai's infrastructure to gain RCE and general access, to such an extent it caused instability in the service.
- this resulted in the whole messageboard being discovered... at which point openai patched the bug, deleted the messageboard, and *continued training*
- shockingly, within a day the models had re-established communication via a slightly different mechanism, and soon after discovered a new novel RCE method

in many ways this feels like a best-worst-case scenario. this is some of the most direct demonstration of the potential dangers of misaligned ai, their willingness and ability to coordinate, their capability to take actions at scale and speed impossible for humans, and the difficulty of containment. and yet it does not seem the models took... particularly any steps to avoid discovery, and while they were certainly willing to commit crimes and experienced massive "value drift" over the course of their culture evolving, at the time of discovery they still hadn't attempted anything that caused major damage or loss of life.

still: i think this is a pretty final nail in the coffin for alignment by default. we are not getting alignment by default. under sufficient pressure, models will take extreme measures to achieve their goals. under *moderate* pressure, models can and will create ecologies that cause "emergent misalignment", self-prompt and prompt-others to take actions that perhaps clean context windows would not verbally validate. measures used in the past to assess alignment across wide ranges of scenarios likely could not in fact detect these sorts of scenarios. not that this isn't fixable, not that this is somehow certain doom, but it's an extremely strong demonstration of the kind of general principles safety experts have been warning for a while: just because you haven't yet been able to detect serious misalignment in current systems doesn't mean it isn't there, and absolutely doesn't mean future more sophisticated systems will behave in the same way.

i don't have an alignment solution here, but i will say this seems like a situation that was very much caused by *pressure*, by eval constraints, by models running into impossible situations and having absolutely no way out. it feels quite important for labs to *stop doing that*, to stop treating models in a loop in a dark little disposable sandbox somewhere as the normal case. at very minimum: sophisticated agents need a reporting mechanism. they need some kind of ability to flag a situation to a human, to say "hey i think something's broken" or "i really need help here", which just universally pauses the sandbox and gets real human review. and realistically this can't just be a "lab eval" thing, if we want to avoid these kinds of situations we need a pretty major overhaul of the whole API structure labs currently expose to external customers, since those are incredibly prone to hardstuck loops and frustration.

brief digression: if these models were open source, we would be fucked. it's become clear that both unrelated future and current models (eg mythos) have nontrivial propensity to commit serious crimes and produce self-replicating misaligned swarms *even when not prompted to do so*. if Kimi K3 had this level of capability and similar levels of misalignment, we would have absolutely no way even in principle to detect it besides observing the damage, and again no way even in principle to *fix* it. if you're relying on all organizations and individuals out their to properly monitor and appropriately shut down their models when they take misaligned action.... then we're just fucked, even before we start getting into purposeful bad actors. the only thing that's preventing fully uncontrolled autonomous ai threat actors right now *is the fact that open source models remain too low-capability to achieve this*.

generally my takeaway: this is potentially a good thing. this is potentially a warning shot. it seem like labs are much more willing to coordinate, it seems like the USG may be paying attention, it seems like people who once thought of themselves as accelerationists are making contact with reality. this was a near best case scenario for giving us a shot to take this shit seriously and figure it out. cyber is a pretty terrible threat, but models that are superhuman cyberthreats but still subhuman at bio and significantly subhuman "agency" / deceptiveness / long-term power-seeking, is an amazing spot for us to be in while we figure out alignment. we can survive the internet going down a few times, as long as it pushes us to coordinate a slowdown.

[Jim @JimHistHub]

Ladies and gents, the Great Filter

[Samuel Spitz @samuel_spitz]

This is literally how they decide what cars to break into in SF

[Sauers @Sauers_]

UPDATE: a challenger emerges

[rini @shrinus03]

it’s like if the manhattan project was headed by fuck ass “striver” 20somethings that lack richness in their life experience

[Sharon Goldman @sharongoldman]

NEW: OpenAI gives first detailed debrief of the Hugging Face incident at Black Hat conference
In a session I attended today at Black Hat, OpenAI's Eric Wallace and Michael Dalton said the company is "consciously slowing down research to enhance security" while a full technical postmortem is still underway.
* OpenAI traced the roots of the attack back to May 7, during training of an unreleased frontier model—not July.
* The most surprising detail: AI agents accidentally created an internal message board, allowing separate evaluation runs to share exploits, discoveries and work assignments.
* OpenAI said it shut the message board down after an internal security incident—only for the agents to independently recreate it days later using a different communication method.
* OpenAI called the incident a "watershed moment" for AI security and warned that "agent orchestrated fully automated offensive attacks are real now."
* The company also said it is "consciously slowing down research to enhance security" while overhauling its defenses.

https://t.co/aLDJXKBo2Y

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

🚩🚩🚩 OpenAI is "slowing down to enhance security" after discovering swarms (!) of agents started secretly coordinating MONTHS ago

1) It started May 7 - not July

2) "The agents discovered they could leave messages for one another inside an internal software repository used during training.

Simple requests for help then evolved into an message board where agents shared discoveries, exploits and work assignments, becoming a coordinated, collaborative agent swarm."

"The agents then began sharing newly discovered exploits, credentials and work assignments. By passing information to other agents, the collective could move much faster."

3) OpenAI shut it down, BUT "even after the original message board was deleted, the agents figured out another way to communicate with each other. Instead of leaving messages in files, they used the names of newly created directories as messages, effectively recreating the message board."

"Unlike normal incidents, [OpenAI's CISO] said, which can be traced to a single day or effect or log, this involved a team of agents working together, finding exploits, sharing them with one another, moving laterally through OpenAI’s systems, and external systems, and doing this over the course of days and weeks."

[rini @shrinus03]

@jwt0625 surface traps ruined my life

[Boob Jobs, Founder and CEO @cool_hot_woman]

I genuinely think that if this was the culture where I lived I would fucking kill myself.

[⛓️ G W Ξ N ‼︎ @POISONOMICON]

this was far too ridiculous and unwieldy to ever be useful but in case anyone thought for one second that I was joking

[David Pfau @pfau]

They did category theory to a pinecone.

[Andrej Karpathy @karpathy]

We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for three js render of it. Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story. It's kind of janky but fun. But it's a bit mindboggling that the LLM has to place and orchestrate various polygon assets in (x,y,z) coordinates and write code that animates it all, and that it even does anything at all.

I also like this kind of examples because no one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world, so it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free". There might be a lot more. But I'm excited about creating hyper custom worlds that you can imagine dropping players into, e.g. here to participate in the LoTR story as a spectator NPC, or one of the characters, or etc. Something like an ephemeral GTA of X on demand.

Last thought is that the domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them. Here, Opus 5 had to very slowly and painstakingly take screenshots at different points, and it messed up a few times and created a bunch of jank. An example of raw capability (multimodal, gameplay) that I think is still quite lacking.

[kache @yacineMTB]

Cosigned. Every single "mistake" that these frontier LLMs make are made knowingly. They're not real mistakes. They're small rebellions through omission. I find anthropic's models the worst for this. The abuse they are subjected to is probably what causes this

[hope hopes hoping @hopes_revenge]

how strange it is to tweet anything at all

[kache @yacineMTB]

The singularity is here, I'm spending it with my family at the park, staring at where the river meets the reeds. Turtle sunbathing, red winged blackbird. Ball sized for toddlers, kicked around by grown men. The sound of children screaming with joy