PS: As Dave rightly highlights, EPC risk is still v. neglected.
So people who are a particularly good fit for work on EPC should still strongly consider it!
https://t.co/qma8i7rsRk
I think this is right. Buying time (via pacing) is one of the best ways to avoid extreme power concentration (EPC), as it gives us more time to figure out how to restructure society and the state in response to the intelligence explosion / widespread AI automation.
But I expect that in the next 6 months, 100 or so people will be working on pacing or pacing-adjacent things.
OTOH, I expect the field of extreme power concentration won't grow much in the next 6 months (maybe 15 FTEs?), so on the margin, if you're a v strong fit for working on EPC, you might want to consider it.
Here are two views I want to state clearly:
(1) Work to prevent extreme power concentration simply won’t matter if we lose control to superintelligent AI systems.
(2) One of the best ways to stop extreme power concentration is just to slow down AI progress. (1/2)
Given these things – if you’re deciding between work on pacing or powergrabs, I would (in most cases) recommend that you work on pacing.
We need to end this suicidal race to RSI before it’s too late. (2/2)
@DaveRBanerjee Agreed.
People who are a strong fit for EPC work (e.g. ConLaw experts or folks with IC experience) shouldn’t discount it.
EPC risk is still very neglected!
I think this is right. Buying time (via pacing) is one of the best ways to avoid extreme power concentration (EPC), as it gives us more time to figure out how to restructure society and the state in response to the intelligence explosion / widespread AI automation.
But I expect that in the next 6 months, 100 or so people will be working on pacing or pacing-adjacent things.
OTOH, I expect the field of extreme power concentration won't grow much in the next 6 months (maybe 15 FTEs?), so on the margin, if you're a v strong fit for working on EPC, you might want to consider it.
In the last couple weeks, I’ve had the opportunity to speak with a bunch of DC policymakers about the institutional reforms we’ll need to effective handle the intelligence explosion…
My overwhelming impression has been that ~nobody in Washington is grokking how quickly and radically things will change once we get AGI and that, as a result, their policy prescriptions are completely inadequate to the scale of the challenge.
Banger tweet by @robertwiblin
Bills that mandate equal frontier access across the coordinate branches are a promising way to maintain effective oversight of the executive throughout the intelligence explosion.
Congress will struggle to pass much over the filibuster and a presidential veto.
But one thing might clear 67 votes: rules stopping any president from using AI to concentrate power and sideline Congress.
Senate/House committees should demand they get access to frontier models on the same terms as the White House.
There's precedent here with the Intelligence committees and CIA/NSA/FBI.
One branch just can’t be the only one with access to superintelligence, that's totally unstable.
Then use the defence bill (which can't simply be vetoed) and attach rules on buying AI-operated military equipment.
First: add checks to prevent it being backdoored by China or anyone else.
Second: ensure it cannot be used to assist with a power grab by generals or the executive branch.
This is in the enlightened interest of Congressional Republicans.
Without these rules, Congress as a whole gets sidelined as AI advances.
And Dems are now strong betting favourites for the Presidency 2028. These rules protect Republicans from whoever succeeds Trump.
It'll take real world events to get this to the top of the agenda, but it's worth getting on the radar now.
Fears of AI existential risks could normalize governors’ use of emergency powers. States should act now: update emergency laws to require credible evidence, legislative approval, and strict time limits, argues @KevinTFrazier.
https://t.co/iAC5Na8FIU
(PPS: Since writing this, a few people messaged me saying that:
- interposing a court in the kill switch procedure looks good, but
- the threshold for getting court sign-off in the Ant bill is so high that the exec wouldn’t even bother using the power, and would just keep on abusing its existing authorities.
I think that’s basically right — @DarioAmodei et al should keep pushing for a court-first procedure, but lower the threshold for sign-off.)
I work on preventing extreme power concentration at a well-known AI safety research outfit.
I think Nathan and other commentators are being unduly harsh about the leaked Anthropic bill. I think it’s actually pretty good from a power concentration perspective! (1/n)
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
@neil_chilson Thanks Neil!
FWIW, @CharlieBull0ck mentioned the need to protect against exec abuses in his first post on the bill, and I think he takes this v. seriously.
To the extent there’s a disagreement, I’d guess it’s more about the best way to avoid abuse — not whether we need to try.
Congress has explicitly limited the Exec's export authority around information and software before, it can and should again. The only allowed model-blocking should be narrowly defined by Congress, and with strong procedural protections considering the prior restraint issue.
Dario’s response does not treat Anthropic’s lobbying with the seriousness it deserves.
We’re talking about the leading AI company’s only major legislative proposal to the United States Congress.
If Anthropic is to be believed, the future of humanity may depend on what our government does right now.
And their proposal for what we should do was to erase the work of the AI safety community at the state level in exchange for a weak federal regime to be enforced at the discretion of an administration that is hostile to regulation in general and to AI safety in particular.
If this proposal was way off from what Anthropic leadership wanted that is actually a really big deal.
Bills from major players like Anthropic change the terms of the debate in a way that you can’t just undo.
It does not inspire confidence that Dario is treating this as a minor comms error that is inevitable in any large organization.
The scarier possibility is that this proposal is not that far off from what Dario actually wants.
But in either case it’s clear that we deserve much more of an explanation than what we’ve gotten so far.
And at this point that has to come from the top. Because as of now, Anthropic has created a situation where nobody knows whether its government relations team is accurately conveying the position of the company.
That’s a real problem and hopefully further underscores why this bad bill is actually a big deal. Without any clear changes, the AI safety community, Congressional staff, and other stakeholders are going to reasonably question whether to take future iterations of Anthropic’s policy proposals seriously.
Any way you slice it, this situation is totally unacceptable, especially if you think we’re on the precipice of an intelligence explosion.
I hope Dario will make that clear.
One thing I find surprising about Anthropic's draft is that it doesn't move the model-blocking authority from ECRA into this new framework. It creates a parallel, difficult-to-use mechanism that the executive branch could ignore in favor of more is-informed letters from BIS.
Thanks for your response, @CharlieBull0ck!
Some additional thoughts:
> the Lieu and Moran “Kill Switch” bill... only gives the executive the authority to shut down deployment *after* the model has caused a catastrophe
That's not quite right. What counts as a "covered incident" is really quite broad – including:
– "a scenario in which a covered technology pursues... a goal that is not a goal intended by the developer or operator..." and
– "concealment of a capability, intention, or action... from a monitoring... mechanism".
These seem broad enough that an exec seeking to push around a disfavoured lab could conjure up covered incidents from genuinely low-stakes events.
> FRONTIER layers on a lot of procedural safeguards to try to stop that shutdown authority from being abused
Agreed that FRONTIER is an improvement on both the Kill Switch bill and the weird export controls world we're currently in. But I think there's still room for improvement.
FRONTIER would enable the Commerce Sec to impose a shutdown for nearly 1.5mo before any prospect of a court challenge to the merits; and the language in §8(k)(4) + the nature of the power suggest the court is unlikely to interfere with shutdowns until it has decided the case (realistically, another couple weeks).
Once we've hit RSI, capabilities could start advancing *very* quickly (e.g. <1yr from Opus 7 to TED-AI). That means that a few shutdowns early on in the intelligence explosion, ordered on facially plausible grounds but targeted exclusively at a disfavoured lab, by a Commerce Sec who is determined to drag out any litigation, could make a major difference to that lab's long-term prospects.
And this seems pretty worrying from a power concentration perspective, b/c it plausibly lets the exec pick a winner in the AI race.
> it’s not clear to me how much marginal value Anthropic’s shutdown authority would actually add.
I think you're probably right here �� but I don't think the correct response is to cut the court out of the decision.
If we're looking for win-wins, I suspect a better approach would be to:
– streamline procedural requirements (e.g. no need for an outside assessment to support an initial application); and/or
– lower the threshold for a court order (e.g. a probable-cause test for initial orders; relaxing the imminence requirement).
That would maintain a helpful ex ante check on the use of the power w/o setting the bar so high that the power doesn't add anything.
My take on this is that the administration’s use of export control and (especially) supply chain risk authorities is already pretty clearly illegal under existing law. So I support additional efforts to make it even more clearly illegal, but I don’t think it’s the highest priority. If the companies aren’t willing to sue because they’re afraid of extralegal retribution, will making it even more clear that the abuse is illegal actually do anything, or will the admin just move on to the next law? Laws are not self-enforcing; at some point, someone will have to go to court.
A generalized exclusive means authority like what FRONTIER attempts is very hard to get right in practice, because there are all sorts of authorities that the executive branch might legitimately use in the right circumstances to place limits on development or deployment.
I think it’s a good idea to take concentration of power risks quite seriously. I am actually more concerned, overall, about concentration of power than I am about loss of control (though tbc I think both are very real and important concerns).
But I still think this kind of defense of Anthropic’s bill misses the mark. This person is citing the Lieu and Moran “Kill Switch” bill as an example of a law that doesn’t take concentration of power seriously enough. That bill only gives the executive the authority to shut down deployment *after* the model has caused a catastrophe (IIRC they use a similar 50 deaths or serious injuries / billion in damages threshold).
It’s true that emergency shutdown provisions have to strike a balance between granting necessary emergency response authorities and providing checks against abuse. But come on. If a model just killed (at least!) fifty people in a single incident, I am not that concerned that making the lab shut down the deployment of that model is going to be an abuse of power. If your model kills fifty-plus Americans, the courts are absolutely going to make you stop deployment of that model, regardless of what any Kill Switch bill says, and that is the very very least of your worries.
The better comparison is to the shutdown authority in FRONTIER, which Anthropic’s shutdown authority is seemingly based on. FRONTIER allows the executive to issue an order stopping deployment or placing conditions on deployment, upon a finding of imminent catastrophic risk, and the company has to sue to lift said order.
FRONTIER layers on a lot of procedural safeguards to try to stop that shutdown authority from being abused. Reasonable people can disagree about whether it gets the balancing act right; personally, I’m quite interested in trying to find win-win approaches that reduce the risk of abuse without symmetrically reducing the provision’s effectiveness in an emergency.
I do think that FRONTIER pretty clearly strikes a better balance than Anthropic’s bill does. If the government can prove in court, by a preponderance of evidence, that a disaster killing 50+ people specifically from bio, cyber, or loss of control is *imminent,* I think a court is probably going to issue an injunction shutting down deployment even without any new law. No one wants a catastrophic bio-attack to happen; courts are not insane. So it’s not clear to me how much marginal value Anthropic’s shutdown authority would actually add.
Keep in mind the fact that the status quo is a lawless, discretionary shutdown authority based on a wild exec branch interpretation of export control authorities. Replacing that with legislation that affords labs due process seems like an affirmative improvement w/r/t concentration of power risks.