Wyatt

Benevolent binary manipulator at @redballoonsec. OFRAK!

[Wyatt @whyitfor]

@psifertex Ghidra sales reps are the worst

[Brian in Pittsburgh @arekfurt]

Cybersecurity has "assume breach" as a key concept.
AI safety/security probably needs a coherent "assume misalignment" concept.

[Thariq @trq212]

the right way to use model capabilities is not to ship 10x more features to prod

it's to spend more time understanding your users, trying experiments, building prototypes, learning about things you don't understand so that you can ship things that actually work

[Martin Fowler @martinfowler]

NEW POST

I see AI and LLMs as bringing a mix of boons and banes, but one feeling dominates: not just are they pretending to be a person, but the kind of person I don't like.

https://t.co/8OUKX5OWde

[Wyatt @whyitfor]

Need to try this out!

[Thomas H. Ptacek @tqbf]

I wrote a thing about this.

https://t.co/D0H7x8rlRl

[Wyatt @whyitfor]

And it will also lead to a renewed push for better alignment.

[Zack Korman @ZackKorman]

Crazy prediction: The next major AI agent incident caused by an AI lab will have been entirely preventable with good cybersecurity.

[J. A. Guerrero-Saade @juanandres_gs]

Third-party evaluators being a handful of random kids from unnamed startups funded by ExRisk donors, with virtually no security experience?

[Matthew Green @matthew_d_green]

There’s some really yucky stuff in this report.

[Wyatt @whyitfor]

1. This is a great read. Happy to be able to see it.
2. The PR aspect, as always, is weird. Should @Microsoft disrupt threat actors using Excel? Should @rapid7 publish similar reports of @metasploit being misused?

[Boris Cherny @bcherny]

The latest Threat Intelligence report is an absolutely terrifying and important read.

As models become more intelligent, without the right safeguards and monitoring they also become more dangerous. Many capabilities are dual use: a model that codes well can be used to hack critical infrastructure; a model that assists with biology research can also be used to engineer the next pandemic.

These issues are complex, thorny, and increasingly important for everyone to understand so that the world can weigh in and respond to rapidly escalating risks.

https://t.co/0rnV2HjPP6

[Wyatt @whyitfor]

Seems it’s still important to know when to buy versus build

[Gergely Orosz @GergelyOrosz]

This is the thing with internal tools: over time, they almost always become subpar vs a tool a vendor builds

At Large Company that have their own, custom o11y, talked with an eng... who said how much they'd love to use Datadog (or similar) because that would "just" work instead of "kind of" work

[_ZN4DionC1Ev @justdionysus]

The cybersec industry focuses heavily on organizational security. I’m increasingly concerned about personal security. Large orgs like Google and Apple can meet the moment but when bespoke exploits for tier 2 products are super cheap, the economics of surveillance are shaken up.

[Dino A. Dai Zovi @dinodaizovi]

Now that frontier AI is getting better than humans at reverse engineering binary code, finding vulnerabilities, exploiting and chaining them, and penetration testing, what should we be presenting at security conferences instead?

This: defenses that scale across vuln classes.

[Wyatt @whyitfor]

Neat!

[Quentin Kaiser @qkaiser]

Secure Boot Bypass on NVIDIA Jetson, it’s time to patch and fuse those keys ! https://t.co/qo2rEtI0Mp

[Wyatt @whyitfor]

@arthurctellis Framing alignment as more relevant than security feels like a false dichotomy.

Alignment is no doubt an important goal to strive towards, but systems (including evaluation systems) should be resilient to and expect misalignment.

[Wyatt @whyitfor]

@mattjay Yup. I literally wrote “hello, world” and fable flagged me

[Wyatt @whyitfor]

The agent revolution will necessarily be televised, celebrated, and dismissed.

[DAIR.AI @dair_ai]

Banger paper from Google.

If you maintain a skill library for your agents, you might want to check this out.

(bookmark it)

This work separates three things that skill-evolution systems usually collapse into one. Raw execution traces, a persistent wiki of accumulated knowledge, and the executable skills themselves.

Experience gets consolidated into the wiki, and every later skill update builds on that wiki instead of on a scattered optimization history.

Ablations confirm the wiki is what carries a lot of the gain. Two results stand out in particular. Smaller models with evolved skills beat substantially larger models without them. And skills evolved by one model transfer across families, where skills evolved elsewhere sometimes beat self-evolved ones.

Paper: https://t.co/6qftGirTpE

Chat with Paper: https://t.co/rrVzkkR1ij

[Rami McCarthy @ramimacisabird]

My thoughtful take on the lab escapes/rogue agents:

[Wyatt @whyitfor]

Remember when decision trees were preferred for some use cases (credit decisions, etc) because explainability was important?

[Ryan Greenblatt @RyanGreenblatt]

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.

I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.

Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.

We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.

Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!

The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.

While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.

In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.

[Wyatt @whyitfor]

Important neologism. But the phenomenon predates AI in other contexts: почтальон (mailman) for bureaucrats in Russia, etc…