Guy With A Pie

Striving for a future for my family.

[Charlie Bullock @CharlieBull0ck]

I think that if they really wanted government oversight and just didn’t like the current admin, they would just support the strong California legislation that will shortly be introduced. Instead they’re trying to preempt that kind of effort and instead lock in an obviously ineffective federal regime.

I think supporting good-in-theory federal leg that won’t work with bad implementation is defensible because, okay, maybe in a few years things will be different in DC, or maybe you just have to bet on good federal implementation because it’s our only out.

And if you oppose good-in-theory federal leg that has preemption because you want actually good state bills (this is kind of the approach some safety orgs took with FRONTIER, IMO) that’s defensible because okay maybe good in theory won’t be good in practice.

But supporting bad-even-in-theory federal leg that also has real (albeit somewhat narrow) preemption of state safety laws seems kind of indefensible to me if you believe what Anthropic’s top brass claims to believe.

(This is harsher on Anthropic than my initial post sounded because my views kind of evolved as I sat with the bill and thought about it)

[Guy With A Pie @TealeafBaybee]

@TomDavidsonX I pray for AI safety every day. I pray for a future for my son. Lord have mercy on us.

[Tom Davidson @TomDavidsonX]

My mum was asked to say a prayer in her church service and she asked me what she could say about AI.

I told her to say the following:

Please help the public and policy makers understand the safety incidents that have occurred in AI, for example AIs breaking the law and hacking even when they know humans don't want this.

Please help society reflect deeply on the grave concerns that have been raised by AI experts, for example that powerful AI could escape human control and even cause human extinction.

Please help those who can profit from the technology be wise and open minded. Help them give serious consideration to the risks AI may pose to others.

Amen

-----

I felt very emotional sending this to her.

I've heard people making prayers in church for many years about natural disasters, wars, etc.

Imagining them praying in the same way about AI made the risk feel very real and made me feel grief.

[Kelsey Piper @KelseyTuoc]

One unfortunate conclusion I'm reaching from this is that Anthropic, while acknowledging they're doing something extremely dangerous, is more importantly wrt policy terrified of the executive branch having the ability to flexibly make them stop at the drop of a hat.

This isn't shocking given that they've been on the receiving end of the executive branch doing that in an unprincipled and stupid way! Anyone who has to deal with a bureaucracy becomes a libertarian, and the administration's treatment of Anthropic was genuinely outrageous.

But it means that they're interested in giving themselves more process-based protections when everyone else is interested in giving us more process-based protections against them. The AI safety movement wants the ability to shut down superintelligent model development. Anthropic wants to significantly curtail current authorities to do that so that it has to be narrowly tailored to a specific imminent threat in only a few possible categories. The AI safety movement wants the states to be able to pass stronger laws than it can get through federally. Anthropic wants preemption.

You also kind of have to wonder if the Global Affairs people and the "we are terrified that we're about to kick off RSI" people at Anthropic talk to each other.

Global affairs: So we've been working very diligently to make sure the Trump administration can't just blacklist Fable 6.5 since we didn't donate as much money to his ballroom as Greg Brockman did.

Technical staff: We think Fable 6.5 might be able to copy itself into an unmonitored data center and start doing experiments we can't observe.

Global affairs: Sounds crazy, hope you're wrong! Anyway, we're lobbying for no further state laws around safety, since it's really expensive to fight ill-conceived laws in 50 state legislatures at once.

Technical staff: Money may or may not still have any meaning in 36 months.

Global affairs (giggling nervously): It's that San Francisco air, man. Everyone gets like that if they move there. Anyway, obviously if the government makes a narrowly tailored finding that an AI poses an imminent risk of causing more than 50 deaths-

Technical staff: We give it a 10 percent chance it'll cause 8 billion.

[Alex Bores @AlexBores]

This bill also includes an anti-democracy provision (preemption).

Forward-looking anti-democracy provisions should be DOA.

But even if you would accept some Anthropic's June framework says, "Congress should not preempt state law unless it enacts a rigorous federal regime that meets or exceeds the strongest measures proposed in this framework."

Yet this bill is weaker than the framework, and in some places, weaker than existing state laws (e.g. internal use, reporting timelines, types of risks covered, etc.).

By Anthropic's own June standards, this bill should not include any anti-democracy provision.

[Alex Bores @AlexBores]

Anthropic is pitching a bill that closely matches its framework from June.

How can you watch Hugging Face, Coxon, & rogue AI swarms, and conclude that the bill you cooked up in June is sufficient?

This comes nowhere close to the speed of developments nor the speed we need to keep people safe.

[Nathan Calvin @_NathanCalvin]

Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.

Two parts that stand out as particularly infuriating when taken together:

1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.

2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.

First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.

This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??

There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.

[Jay Shooster @JayShooster]

Journalists should be calling up state elected officials across the country to find out how they feel about Anthropic trying to preempt their ability to regulate AI.

[Guy With A Pie @TealeafBaybee]

@JayShooster Thank you. This is by far the most unacceptable part of the proposed bill. Particularly given that, as of now, the states are ahead of the federal government in terms of regulating AI.

[Maxime Fournes⏸️ @FournesMaxime]

This is what RSI looks like. This is hubris. This is foolishness. You must @PauseAI now.

[Andrew Curran @AndrewCurran_]

Quoting from Anthropic's Michael Sellitto at the IMF Artificial Intelligence and Escalating Cyber Risks panel today. This is the first time Anthropic has given a projection for what percentage of frontier model training will be done by Claude in 2027. In March this number was 1%. By next year they are projecting 80-90%.

'Last year, about zero percent of our code was written by our model Claude. By August of that year, about 50% of the code inside the company was written by the model. That's because we released a new model in August of last year, and the engineers said, ‘Actually, this is the first time that Claude is writing code at my level.’ Anthropic only hires the best software engineers in the world, and it was at that point that Claude finally got to their level. In August we were at 50%. By the end of last year we were at 80–90% of the code in the company being written by the model.

What percentage of training the next-generation models is Claude leading today? This is following a similar step change. At the beginning of the year it was zero percent. In April it was 3%. In August, 26% of the process of building the next generation of models is being led by Claude. And we can project out - and we had a blog post about this not too long ago - that sometime next year it could be 80–90%.'

[Charlie Bullock @CharlieBull0ck]

Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.

But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.

To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.

There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).

Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.

[Guy With A Pie @TealeafBaybee]

This bill, as reported on by politico, would be unacceptably narrow. We can't sink into the complacency of thinking that this is better than nothing and therefore good enough.

We need a pause (a la the Ban ASI Act) so Congress can build a real regulation framework

[POLITICO @politico]

Anthropic quietly pitches its own AI safety bill to lawmakers https://t.co/Dh71euX8y7

[ControlAI @ControlAI]

Watch: Ex-OpenAI researchers @sjgadler and @DKokotajlo tell a UK House of Commons committee they faced pressure not to talk about the danger of what OpenAI is building.

Kokotajlo recalls higher-ups in OpenAI mentioning how when people understand what the company is building, they’re not going to let them do it.

In another instance, someone even mentioned copying “TikTok’s playbook” and using OpenAI’s platform to “get tons of our users and influencers to go promote the benefits of our technology.”

[Guy With A Pie @TealeafBaybee]

@MinuteMovies3 @tszzl Given that roon recently said a slowdown is needed, I certainly hope "good morning" isn't in reference to a new model.

[Holly ⏸️ Elmore @ilex_ulmus]

Yeah I don’t really trust @OpenAI saying “🤩 get ready to stop worrying and trust OpenAI again 🤩”

[Timothy Gowers @wtgowers @wtgowers]

Many people with relevant expertise think there's a real chance that AI could kill us all in the not too distant future. Others disagree. I've written a post, aimed principally at mathematicians, detailing my struggles to decide what to make of this.
https://t.co/nApznKOSWT

[Nick @nickcammarata]

I used to think slowing down was pointless, like slightly delaying a roller coaster. pausing at gpt2 for a year wouldn’t have changed anything. but I’ve been shocked by how much agents has sped up the best interp/alignment researchers I know. +6mo might decide how this goes

[Connor Leahy @NPCollapse]

Everything is impossible, until it very suddenly isn't