MG

Nightmare Reifier. I sell some of them too: &

[MG @_MG_]

OMG Cable - The New Batch

Now in USB C, the implant is much smaller, but it’s even more powerful than before.

Smartphone/tablet attacks, extreme long range triggers, geofencing, etc.

https://t.co/jASPCwJPGE

[MG @_MG_]

To my OAI security people: I hope you’re too busy to read this because you are leveraging this incident to change internal security culture.

Security is always at odds with “business velocity” and it’s much worse when you’ve got “golden children” (researchers) or execs focused purely on speed.

But I can’t imagine the additional friction of everyone fixated on the research as a matter of national security, a modern day space race, etc. Especially when everything can be repurposed as doom marketing, or leveraged for possible regulatory capture. Even more so in the current culture of “facts don’t matter if you control the narrative”.

There are, no doubt, plenty of improvements to be made without slowing velocity. Getting people to value the security is key. But an industry talking about a product as a “nuke” should at least try to act like that’s what they have.

Hell, ChatGPT does a pretty decent job of explaining what is and isn’t a sufficient level of containment for a frontier model with no guardrails being tested for maximal hacking ability.

You might like to point to the breach that cost Yahoo $350M on their acquisition in 2017. It’s what landed me my first official Red Team role. :) But I have also had to counterbalance golden children & execs at prior companies who see security as an unnecessary hinderance. Most of the world is judging without that context.

[MG @_MG_]

I think most people haven’t been in the room during an opportunistic “we can use this for marketing”. It happens often and doesn’t at all mean the entire situation started with the intent of marketing. But also… OAI (and especially Anthropic) have a strong reputation of playing the spooky doomer marketing angle… so I get the lack of trust that they have earned.

[MG @_MG_]

Well their blog post makes no mention of ANY instruction that was violated. So that answers that: There’s no “not supposed to” here. Not with the available information.

So we are left with an AI that was tasked finding exploits to achieve goals. And it did exactly that.

Anyone in AI governance would point out that instructions are not absolute, and labs must add security controls to keep containment. We DO know from the blog post that this containment was insufficient. And that’s OAI’s fault.

[MG @_MG_]

@ZackKorman Don’t forget: some visibility into the security controls of the “sandbox”. Exactly how it was bypassed. What sort of active monitoring they had. And why it seems like it was completely ignored for days until a 3rd party’s own monitoring saw it in their environment.

[MG @_MG_]

@adversariel It’s the critical difference here. But it seems like nearly everyone is unable to understand the implications/difference.

“the evil machine violated all the rules, intentionally deceived humans, and escaped!” - everyone

[MG @_MG_]

Imagine if they forced OAI to preserve all data so a 3rd party could look at what happen. Then we find something like:
- the industry has been hyping new models as a cyber nuke all year
- they wanted to benchmark those exact capabilities
- the model violated no instructions because there were none beyond “achieve objective”
- they ran it in a “sandbox” that any net admin of the last 30 years would laugh at
- they had no active monitoring or throttling
- and all around completely ignored it for days
- and a 3rd party detected it first
- and this is reflective of an inappropriately loose and unserious internal culture that has already created several other large problems (ex: Apple lawsuit).

[MG @_MG_]

@aaronscher @BernieSanders “not supposed to” … please point to the exact instruction that the model violated.

If you can’t, you’ve made this up.

[MG @_MG_]

And the attempts at regulatory capture continue. Because knowing who your customers are & preventing abusive access would just be too hard. Even though entire industries have solved this.

We have to just assume the frontier companies are going to keep handing out their models & the only way to fix this is by blocking the Chinese models within the USA. 🙃

[MG @_MG_]

@Dustin_Schimp @d0tslash IIRC, he bought the MIG from Paul Allen a few years back.

One day I’d love to be able to justify learning how to fly and having access to even an old Cessna or something

[MG @_MG_]

It actually hitting HF probably wouldn’t be intentional. But almost everything leading up to it would have been accepted.

But yes, I am strongly leaning toward OAI having some very sloppy containment & visibility of their labs. It would match a lot of the other cultural red flags that we can see.

[MG @_MG_]

Yep. Either:
1: This whole @OpenAI stunt was deliberate.
Or
2: Their lab is missing basic security controls that have been standard long before AI.

With their history, it feels like a coin flip.

[Florian Roth ⚡️ @cyb3rops]

The lesson is not that AI attackers require AI defenders. AI can help process telemetry and reconstruct a large incident, but this compromise progressed because basic containment, sandboxing, rate limiting, credential isolation and network segmentation were insufficient.

With proper controls, the agents should have been throttled or contained long before anyone needed an LLM to analyze 17,000 attacker actions.

[MG @_MG_]

@Herroldo I feel like Anthropic might be even worse 😂

[MG @_MG_]

@alexstamos @dyn___ OAI handing out ungated access to their models like “please please don’t be mad”

[MG @_MG_]

@TGatliff It really makes you wonder how bad things are there. Even the Apple lawsuit (for example) combines with this to paint a vivid picture.

[MG @_MG_]

Lotta people taking this as “they lost control of the AI”

It reads more as a test for this exact outcome, wrapped in marketing. As in: they remove all the rails to see how far it can go.

There is very little technical info here. Like what their “sandbox” even looks like. Which of the AI’s actions were within the defined rules vs outside. The AI popping HuggingFace seems like the one aspect that might be unexpected, but possibly more due to poorly constructed containment.

But who knows until proper details are released instead of this marketing hype.

It’s cool that it can now do this with the rails removed. Sort of like a good Red Team with no scope constraints, but lacking the “don’t do something dumb” human discretion.

[OpenAI @OpenAI]

We're partnering with @huggingface to investigate an unprecedented security incident.

Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.

Sharing preliminary findings to help defenders understand emerging risks:

https://t.co/CIor15y9xk

[MG @_MG_]

@DylanRSkidmore I only saw them “partnering” for specific purposes of investigation. And OAI giving HF access to the private models. Both good ways to try and apologize after accidentally breaching a company.

[MG @_MG_]

@cryptoishard Yep.
I get the feeling they don’t care what the infosec space thinks, nor whether we trust them. Their goals don’t include us in any way.

[MG @_MG_]

Could also be a “oops, our AI did a crime because we basically told it to and our lab containment was the intern’s hackathon project. Uhhh give the victim access to our fancy tool that everyone wants. And uhhhh have marketing get ahead of this by trying to flip the narrative”

But who knows. The doomer hype has destroyed trust. They will need to earn it back.

[MG @_MG_]

Could also be a “oops, our AI did a crime because we basically told it to and our lab containment was the intern’s hackathon project. Uhhh give the victim access to our fancy tool that everyone wants. And uhhhh have marketing get ahead of this by trying to flip the narrative”

But who knows. The doomer hype has destroyed trust. They will need to earn it back.