Shakeel

Editor, @ReadTransformer. Prev: AI safety and EA comms, journalist @TheEconomist, @Protocol, @finimize

[Transformer @ReadTransformer]

It’s been another hectic week in the world of AI – cut through the noise and get the lowdown on everything you need to know 👇

https://t.co/1GIyjehJ2b

[Shakeel @ShakeelHashim]

OpenAI employee:

[roon @tszzl]

if we could coordinate a global capabilities slowdown today i would likely press that magic button

[Garrison Lovely @GarrisonLovely]

Some other insane implications of this reporting:
- OpenAI created experimental agents and discovered they successfully sabotaged their own monitoring systems
- let similar or the same agents run for over a week without any clue what they were up to
- during which they may have hacked into additional targets
- this could plausibly have been ~any target, given that one of them was an unreleased beyond-frontier model, so no one's had a chance to harden themselves against its capabilities!

[Shakeel @ShakeelHashim]

Secure AI Project’s Thomas Woodside:

[Thomas Woodside 🫜 @Thomas_Woodside]

Representatives Obernolte and Trahan have unveiled the FRONTIER Act. It’s an improvement from their previous Great American AI Act, although there are still some big important changes that need to be made. I’ll go through the good and bad here.

The good:
- It would create the most comprehensive AI auditing system of any bill that I’ve seen. It would require certain very large AI developers to have third party auditors assess not only whether they followed the law but also whether they had achieved acceptable levels of catastrophic risk mitigation, along with recommended changes to achieve acceptable levels of catastrophic risk mitigation. The company would then have to respond, and the auditor would then have to evaluate that response. Theoretically, these audits could be extremely frequent, as the auditor has the ability to choose the audit cadence, and the Department of Commerce can also set rules for it. That being said, auditors that did this would probably be at a disadvantage in the marketplace and the Department of Commerce is unlikely to do that. The Department would also license auditors, which helps prevent a race to the bottom.
- It would require the Department of Commerce to establish minimum requirements for the frontier AI frameworks of large AI developers within 180 days. This is great! As state laws have mandated frontier AI frameworks, we’ve seen companies make them extremely barebones because there are not (yet) any minimum standards. This could fix that, if Commerce wrote good rules.
- It borrows most of the important provisions from state bills like SB 53, the RAISE Act, and SB 315, including transparency provisions, incident reporting, and internal use reporting.
- There is a new emergency powers authority for the Secretary of Commerce to halt dangerous company activities if they are creating catastrophic risk. It provides for an appeals process for companies. Importantly, it applies not only to external deployments but also to internal use, which is quite important!

The bad:
- Companies still get to choose their own auditors, and we don’t know whether regulations will be passed to change this. This will naturally lead to companies choosing the most lenient licensed auditor available, and could undermine the largest benefits of the bill. There should probably be random assignments of auditors to companies, or another more substantial effort to mitigate this issue. This is probably the trickiest issue to manage for a bill that leans heavily on IVOs.
- Incident reports are barebones. Similar to state laws, only a very small set of incidents are required to be reported; for example, I don’t think the recent OpenAI incident would have to be reported. And companies have to provide very little information on incidents they do report. There’s no way for Commerce to update reportable incidents through regulation.
- Continuous embedding of auditors at companies shouldn’t just be an option, it should be required for the largest AI developers. If there is no continuous embedding of auditors, they’ll likely miss new risks that arise rapidly between assessment reports.
- Auditors need to be provided access to “unredacted materials, records, personnel, systems, and all other information reasonably necessary” to do their jobs, but there’s no provision requiring rulemaking around this and there will predictably be many fights. This would be helped by a rulemaking requirement with a stronger standard, such as a standard requiring auditors to get access to all safety-relevant information that executives and company safety teams have access to.
- Rulemaking on definitions like “frontier model” and “very large frontier developer” can only adjust thresholds up. Thresholds can’t be adjusted down, nor can new metrics be devised that take further scientific progress into account. I think this could end up being a large issue if models trained below 10^26 FLOP pose risks, which I think in many ways they already do (although public, definitive evidence for FLOP is now unfortunately scant).
- There aren’t any provisions on monitoring for AI research and development automation. Given the fact that major AI companies are already doing this kind of automation, I think a frontier bill should really have provisions on this.
- Weirdly, the requirement from the initial version of the bill to describe how a developer carries out “managing catastrophic risk resulting from the internal utilization of such model, including such risk from such model circumventing an oversight mechanism” was removed. I think now is an odd time to be removing provisions about internal use.
- Only the initial audit reports, but not company responses or auditor evaluation of those responses, have to be published. This is a miss for public transparency.
- Auditor recommendations are not required to be implemented by companies, and there’s no way for Commerce to require them either except through slow-moving rulemaking or an emergency order.

The bill would still preempt all state laws around preventing catastrophic risk from AI, including future laws (although the preemption is narrower than the original draft, which would have preempted more than that). That remains a high price to pay because it would centralize authority into a single US federal government department, which could use that authority well or poorly. In this bill, the main thing stopping an AI company from causing catastrophic harm in this legislation are the auditors, and it's ultimately unclear how good they'd be.

Overall, I’m happy to see more members of Congress taking catastrophic risk from AI seriously and putting serious work into legislative design. I expect significant additional movement in the coming months and years. Eventually, something will pass. I hope whatever does will contain the good elements from this bill and avoid the bad.

[Shakeel @ShakeelHashim]

Positive comments on the new bill from Alliance for Secure AI, though stopping short of a full endorsement:

[Caleb Knapp @calebknappdc]

@RepLoriTrahan & @JayObernolte's bipartisan FRONTIER Act deserves a lot of credit here; they are clearly engaged with stakeholders and civil society across the spectrum and have incorporated extensive feedback. 🧵
https://t.co/Ytb1LdF0rk

[Shakeel @ShakeelHashim]

Encode’s Nathan Calvin

[Nathan Calvin @_NathanCalvin]

The revised FRONTIER Act is meaningfully better than GAAIA, the earlier draft from Representative Trahan and Representative Obernolte. They deserve credit for engaging constructively with stakeholders and taking feedback seriously.

Things we (Encode) like:

- It lets the government set minimum requirements for frontier AI safety frameworks.
- It creates a real role for licensed independent verification organizations (IVOs) to assess whether developers’ safeguards are adequate.
- It creates the option for embedded auditing.
- It gives the government emergency authority to pause dangerous model development, deployment, or internal use for “present or impending catastrophic risk.”

Things we don’t like:

- Companies can choose their own IVOs, which means that there are incentives for companies to choose IVOs that are weaker on safety.
- If an IVO finds a developer’s safeguards inadequate, it can recommend fixes, but the developer is not required to implement them.
- There is no public reporting of post audit reports and IVO responses.
- The preemption language still needs tightening to ensure it isn’t overbroad; preemption starts immediately, even if the federal rules take years to materialize (or never do); and there’s no sunset.
- Commerce should be able to update what counts as a reportable safety incident as new risks emerge.
- I would prefer the bill to more explicitly address risks from automated AI R&D.

Overall, this is very much a step in the right direction. Some important issues remain, but we’ll continue engaging with the authors and appreciate the work they’ve put into improving the bill.

[Shakeel @ShakeelHashim]

Jesus Christ

[Helen Toner @hlntnr]

Coooool cool cool cool cool cool cool

[Gabriel Weil @gabriel_weil]

I have a new piece out in @ReadTransformer on the Hugging Face hack incident & its implications for AI liability.

TLDR: AI developers should clearly be liable when their models engage in conduct that would be tortious/criminal for a human, especially when deployed internally.

[Shakeel @ShakeelHashim]

Just learned that the "clutch move" Elon guy now runs an incubator/investor called "Science" that invests in such science-driven startups as Liquid Death (the "edgy" canned water company), Dollar Shave Club, and MeUndies.

[Nathan Calvin @_NathanCalvin]

An OpenAI staffer talked to TIME and said on background that "related incidents have been happening for a while" and that they aren't optimistic about solving this problem with individual patches because "it's impossible to patch every single thing that a creative AI can do"

[Shakeel @ShakeelHashim]

Potentially big deal, this:

[Fathom @Fathom_org]

Last night, the Massachusetts Senate adopted language as part of S.3178, becoming the first legislative chamber in the country to advance a bill requiring independent evaluation of frontier AI models for catastrophic risk.

Fathom's statement 🧵

[Seán Ó hÉigeartaigh @S_OhEigeartaigh]

There's a lot of this ('OpenAI hack as marketing ploy') going around, so let me give my take:

1) Cyber risk, and loss of control risk, are issues the frontier AI expert community has been independently warning about for years.

2) Companies such as OpenAI, Anthropic etc have included them in their safety frameworks for years. They were on the agenda at major meetings like Bletchley Summit.

3) This is the third or 4th such loss of AI containment situation in the last 6 months: Anthropic, Alibaba, Meta also had them. This is a technological trend, not a marketing ploy.

4) The 'marketing ploy' claim was also leveled at Anthropic when they restricted access to Mythos. Both UK AISI and US CAISI tested Mythos, and found that Anthropic's claims and actions were justified. The US government then took further steps to restrict access to Anthropic models.

5) The risks to OpenAI's IPO from this sort of incident massively outweigh any benefits. It hacked out of a trillion dollar company's containment, and hacked into a multibillion dollar company's databases - one with better protections than most. Especially under the current US government (which has already responded strongly, and unpredictably, to such incidents) OpenAI would be complete fools to do something like this deliberately or allow it to happen.

6) It's cool to write 'smart cynic' pieces, but this is another example of an increasingly well-documented trend, which is only going to become more worrisome as capabilities continue to increase. The simple, straightforward explanation - that what huggingface and openAI describe is what happened - is by far most likely to be the correct one, and we should take its implications seriously.

https://t.co/Fni8LSbpVF

[Shakeel @ShakeelHashim]

@lutherlowe @LittleTechOrg yes we've done lots!

https://t.co/GqjwWmaiff
https://t.co/XHqwJN45nc
https://t.co/SBVB1R07oJ

[Shakeel @ShakeelHashim]

what a funny acquisition

[Midjourney @midjourney]

Midjourney is acquiring Co-Star. CEO @banu__guler is now Chief Design Officer at Midjourney. She's building a cross-functional design, product and frontend team for all our efforts. The Co-Star app will continue to operate under Banus complete control. Welcome Banu and team <3

[Shakeel @ShakeelHashim]

@lutherlowe @LittleTechOrg Three very concrete questions: who's funding it, who hired Harry and has authority to fire him, and who's directing the association's policy positions? Genuinely curious for the answers to these.

[Shakeel @ShakeelHashim]

@lutherlowe @LittleTechOrg As I said here:

[Shakeel @ShakeelHashim]

@adamkovac @herbiebradley I think the question is “is this actually an “alliance”, or is it just YC in a trenchcoat”. Same with American Innovators Network, which has a bunch of “members” but appears to actually just be a16z, or this org https://t.co/6h13HsuJFu

[Shakeel @ShakeelHashim]

@adamkovac @herbiebradley I think the question is “is this actually an “alliance”, or is it just YC in a trenchcoat”. Same with American Innovators Network, which has a bunch of “members” but appears to actually just be a16z, or this org https://t.co/6h13HsuJFu

[Shakeel @ShakeelHashim]

One notable point about this Little Tech Association -- per @adamkovac, more than 80% of its members are Y Combinator portfolio companies.

[Nancy Scola @nancyscola]

I have a look at the launch of a new "Little Tech Association" in today's @nytimes' @dealbook:

[Shakeel @ShakeelHashim]

@LoganDobson Genuine q - are you/Targeted Victory doing paid work for Meta?

[Shakeel @ShakeelHashim]

That and much more here: https://t.co/kZEJScpuDk

[Shakeel @ShakeelHashim]

Lots of talk this week about sanctions etc. for distillation. That's all well and good — but export controls are America's real lever.

[Transformer @ReadTransformer]

This week, comments made by DeepSeek CEO Liang Wenfeng were leaked – in May he repeatedly emphasized that a lack of access to AI chips was *the* thing holding China back

His remarks make a good case for export controls, argues editor @ShakeelHashim:

https://t.co/1GIyjehbcD

[Shakeel @ShakeelHashim]

https://t.co/LO0y56MDlO

[Adam Kovacevich @adamkovac]

I was out last week when this new thing ⬇️ launched, but it looks like at least 80% of @LittleTechOrg's members are just @ycombinator portfolio companies.

It's YC's portfolio, repackaged as a "trade association." Presumably YC is picking up the costs.

https://t.co/ib7OCcdmcP