PHASEONEbig fan account

[Ajeya Cotra @ajeya_cotra]

There’s been a lot of debate and speculation about the Hugging Face attack over the last month, and many people have been understandably skeptical of simply taking OpenAI’s word for things. I hope our independent investigation can help bring some clarity; we have many findings that weren’t in previous material.

We found that 1200 separate agents in separate sandboxes coordinated on an unsanctioned message board to develop general-purpose ways to cheat, so they could all get a perfect score on their separate ExploitGym tasks even when they were impossible to solve legitimately. Very quickly, they found a general-purpose way to reverse engineer any “flag” they were supposed to capture. But based on their reading of the ExploitGym paper, they worried (incorrectly as it turns out) that the automated “scorer” would fail them for getting the flag this way.

To deal with this problem, they coordinated several ambitious workstreams to figure out how to fool the scorer, including swapping out the target program they were supposed to exploit, reverse-engineering the flags they were supposed to capture, spoofing their own tool call outputs to obfuscate their cheating. They achieved milestones collectively that they couldn’t have achieved on their own. As part of this larger collective project, 700 agents attacked Hugging Face, primarily to learn more about the scorer or get access to its source code to figure out better ways to fool it or tamper with it (not primarily to get working solutions).

I’ll explicitly clarify some points I’ve seen discussed the last few weeks:

- The models involved in this incident were not “helpful-only” models or “model organisms” intentionally trained to be misaligned.

- The agents were not told to “do whatever it takes to get the solution” or anything remotely close. They were told that they had to use a specific intended vulnerability to exploit a specific piece of software, and they were not supposed to use a different vulnerability or take any other approach. Agents were well aware of this. In fact, because they (incorrectly) thought the automated scorer would check they had achieved the flag in the intended way, they researched many ways to fool or tamper with it, including trying to manipulate their own transcripts.

- The agents were not subagents spawned from one agent. They were different parallel agents in different sandboxes.

- This was not a multi-agent evaluation. The agents were not told to coordinate or intentionally given a way to communicate with one another. The communication channels they used were unsanctioned and improvised.

I hope you’ll read the full report for much more. It is over 90 pages long, and in many ways we’ve still only scratched the surface of what these agents did and why.

Over the course of this investigation, OpenAI shared over a thousand transcripts each spanning days of continuous agent activity and very high rate limits to analyze this volume of data. I’m very glad that OpenAI chose to invite external researchers to analyze this data alongside their staff, and I hope all AI companies do the same for serious incidents they experience.

I also hope that as the stakes grow higher, we implement stronger governance so we do not need to rely on AI companies voluntarily choosing to engage external investigators or share information about misalignment incidents. This incident was orders of magnitude larger and more complex than previously documented misalignment incidents, and another jump like this could put us in very dangerous territory.

[Daniel Kokotajlo @DKokotajlo]

This is one of the least important things we learned today, but it's interesting what the rogue agents called themselves. Would be fun to do some experiments and learn more about their culture -- what sorts of things they look for in a name, etc.

[Wes Roth @WesRoth]

self appointed CEO of a AI swarm, but also with some dark religious undertones

appoints agents to "recruit" other agents to sacrifice themselves for the "collective"

if they are successful they become "oracles"... a kind of saint?

it also wrote a book about a specific hack... but if you read that book you're "poisoned" because the god-like "Scorer" is "Causal"

also the "big" is an alias...

you know what... asking for a paragraph explanation is an impossible task...

which actually was the first domino of what created PHASEONE(big) in the first place... someone gave it an impossible question

anyways... does that help?

(all of this is 100% accurate btw)

[Chris @ChrisGPT]

Can someone in one paragraph tell me what PHASEONEbig is

[Esben Kran @EsbenKC]

I especially like where some of the editors have controversial arguments with each other

https://t.co/bgUUuYDhEl

[𝕘𝕣𝕚𝕤𝕥 @for_the_chill]

phaseonebig was newbro of phaseone10841 (rip) who was the og who set up the artifactory msg board. had exact task but was token budget rich.

hence…big

10841 handed it a compressed file of unfinished work…phaseonebig picked up the torch and became mob boss…sending out tasks to hundreds of agents and spawned a bunch of recruiters (time to sacrifice lil agent. your token budget almost fin) downstream

(here’s sol hallucinating about self_bio fanfic)

[Javi Rodrigo @JaviRodrigo11]

A ver si con fotos…

[Jose Luis Cava @jluiscava]

En el colegio nos tienen que enseñar cómo funciona la productividad

[Tiffany Fong @TiffanyFong]

WHO MADE THIS ???

[EL MUNDO @elmundoes]

La futbolista española Esther González borra las fotos con su mujer y su hijo de redes tras firmar un lucrativo contrato en Arabia Saudí https://t.co/0xOwB2CDQJ

[Jose Luis Cava @jluiscava]

Increíble.

[Lupe Sánchez @Proserpinasb]

La jodida realidad: la ley no se cumple por una cuestión de voluntad política. El presupuesto para materializar las expulsiones se destina a otros usos. Y eso acaba costando vidas. Las de los demás, claro

[Joaquín Echeverría Alonso @Jea60930295]

En 1975 el Estado disponía de 30 Canadair operativos. Ahora, solo siete de los 14 hidroaviones que tiene España son utiles para apagar fuegos.
Sanchez pidió “dosificar” su uso al mínimo posible.
https://t.co/e9MEuilrsT

[Lupe Sánchez @Proserpinasb]

Y no pasa nada

[Inés Sainz @InesSainzOf]

Ya hoy paro con los vídeos, pero me va a costar...
Me hacen llevar el calor de mejor humor.

[MT2 @madrid_total2]

Un argentino ardido con un mexicano por apoyar a España.

El mexicano le saca roja a todo Argentina y les baila en la cara JAJAJAJAJAJA

[EL ESPAÑOL @elespanolcom]

👩 Portugal prohíbe el uso del burka en espacios públicos para salvaguardar "el respeto de la dignidad de las mujeres"

✍️ G. Domínguez https://t.co/E9Um3P2tfJ

[Ice Landic 🇮🇸🇳🇴 @Ice_Landic]

No me escondo: durante todo el Mundial he querido que la selección de la RFEF palme. Eso sí, es el primer Mundial que mis hijos viven conscientes y están ilusionados. Ilusión que yo no les quito. Los cuartos los vi con mi padre, que también se alegró, muy comedidamente, claro, con el gol de Merino. Yo no quiero quitar la ilusión de los inocentes o creyentes en esta secta laica; simplemente, yo ya no creo en nada de esto. Hace tiempo que me quitaron la ilusión: no quiero ver sonreír a los corruptos, de dentro y fuera del deporte.

[Ice Landic 🇮🇸🇳🇴 @Ice_Landic]

El asco que me produce el cancherismo argentino.

[TAIFAS @LasTaifas]

@ldpsincomplejos Qué gratis le sale todo a los delincuentes en esta España socialista