Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Just read a Chinese article calling this "the darkest moment for Chinese AI." Seeing pushback in the comments, but basically it lays out the bear case quite extensively:
1. Getting compute is harder whichever way you go. Buying advanced NVIDIA chips is difficult, renting GPUs overseas is coming under pressure, and proposed restrictions on chipmaking equipment could make domestic alternatives harder to produce too.
2. Catching up could get a lot more expensive. If learning from leading models’ outputs aka distillaion becomes harder, Chinese labs have to spend more on their own data, experiments and training. That gets uncomfortable if investors are also losing patience. Independent labs don’t have an advertising or e-commerce business to pay the bills.
3. Robot-brain startups could find their advantage disappearing. Frontier labs are moving into robotics, potentially eating into what specialist models can offer. Meanwhile, robots still have to make economic sense against China’s relatively low labor costs.
4. More users don’t necessarily mean enough revenue. China has huge digital audiences, but getting people and businesses to pay sustainably for AI is another question. Everyone is competing on price, American models are getting cheaper too, and research isn’t getting cheaper.
5. Even the areas where China already leads face risks. Optical-component suppliers are benefiting from AI infrastructure spending, but proposed US restrictions could threaten access to major customers.
That’s a lot hitting at once. The worry is that companies keep making better models while needing more money to stay competitive, without earning enough to fund the next round.
Now, it feels like every six months it is a "darkest moment" for Chinese AI. But I think these are real concerns. We'll focus on this topic at @techbuzzchina (and present some hope, I think) next week.
The U.S.-China AI dialogue appears to be moving in a good direction. Last week, Secretary Bessent announced that the U.S. expects to exchange best practices for pre-release testing and evaluation of models. This would represent a meaningful step up from the September agreement, when the two sides agreed to establish an information sharing channel. The key question now is how to do the best practices exchange: what experts will participate? How detailed will their sharing be? Will it go beyond testing toward post-deployment safeguards?
From Scott Bessent earlier this week:
The US and China will establish a new AI dialogue with China to guard against risks, exchange best practices for testing and evaluating models, keep humans in control, and alert one another of serious incidents.
Kissinger smiles from heaven
Excited that this report on implementing California's AI safety regulations (plus recs fur future policy) is finally out!
I wrote the assessments and evals section, highly suggest having a read if you're interested in how to improve third-party assessments :)
Very glad that this report is out, and to have contributed to it. I'm biased, but I think it has some of the most serious, careful analysis of how frontier AI policy can work in practice and where it should go next.
https://t.co/SIutHJOaHL
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
It's still very hard to imagine any AI legislation actually passing in the next two years. The White House clearly doesn't seem to be on compromise course, and I suspect a lame-duck President clinging onto power and legacy will be inclined to aggressively wield the veto.
The House, fueled by an anti-corruption and anti-AI mandate, will pass a billion things; the Senate is too blue to really temper these bills to survive the veto, but not blue enough to actually pass them against the administration.
The narrow path to a veto-proof majority, or even to 60 senate votes, for an AI bill runs through one of two channels: votes from natsec-concerned moderate Republicans (who are still not all that likely to outright break with the admin) or from the anti-AI populist wing (who are at risk of diluting any narrow frontier bill into a controversial anti-AI catch-all).
It's hard to imagine how that kind of fragile coalition just emerges from a neutral position in the next few months. Yes, maybe, if VP Vance and political advisors realise the risk of leaving the AI flank open until '28, they could lean on the White House to enable some compromise - but decisions like the Miller appointment should make you much more doubtful that moderation will be the play.
Realistically, the most promising policy window is in response to the next crises. That's a common refrain: we write a bill, put it in the drawer, and if something big happens, we'll pass it immediately. It's particularly tempting because it allows advocates to ignore political reality and instead write down their favorite bill, hoping that exogenous shocks do the political lifting.
I think it's a bit harder than that: you need to draft legislation that makes sense to policymakers now so that it even goes into the drawer - or more accurately, that it gets influential sponsors and maybe a markup so that it's sufficiently red-teamed to pass in a pinch. But it also needs to be ambitious and specific enough that it seems like a solution to whatever new problem emerges. Congress might not reach for cyber bills after bio incidents, or for common-sense low-hanging fruit after major loss of life.
Since we don't know what the hypothetical wake-up call looks like or when it'll arrive, I think that means a lot of hard work: getting enough different legislative vehicles ready that we'll have one ready to go no matter what happens, but have these efforts be sufficiently coordinated that they don't compete for immediate attention here and now.
That kind of positioning is very different than introducing one big AI bill and feeding it into a pipeline that inevitably ends well before being signed into law. If we don't get this right, I expect a lot of posturing in the short term as we ultimately fail to pass a reasonable bill, immediately superseded by unanchored chaos as soon as the next incidents hits and we don't have a suitable bill developed far enough.
The odds of passing legislation have probably gone up over the week, but they still hinge on tactical preparation and bipartisan circulation in advance of very specific policy windows. That won't always align very well with the current vibes of accoubtability enthusiasm and (however well-motivated) congressional reconquista.
Carnegie's @Scott_R_Singer explains why Washington's lack of AI safety legislation undermines the US's ability to pressure China on AI safety:
"The overarching question, if you care about US-China relations, is the US gonna govern AI domestically?"
"If you have a conversation with the Chinese and you say, 'Hey, you all should really do more on AI safety,' the first question that they'll reasonably ask is, 'Well, why isn't Washington doing more? If you really believe sincerely in the risks, then why is there no national legislation?'"
"You can talk about what's going on in California. You can talk about all that the companies are doing voluntarily. But it really just doesn't totally land."
@CarnegieEndow
Absolutely right, and to AI specifically, the public isn't going to tolerate substantial job loss, the prospect of getting decimated by an AI-enabled bioweapon, or whatever else because of some vague notion of "China." It just doesn't land when the cost of living is high and people have families they want to take care of. People need to feel the benefits a lot more than they do now for there to be a robust techno-accelerationist camp (and FWIW I think it is good for the discourse for there to be one! But they need to make their argument much more accessible to a public that is afraid and angry at the tech.)
1/ The US midterm election outcome wasn't about China, but there is a lesson in it for the forthcoming China policy debate in 2028. When the US public is not brought in or bought in to major foreign policy actions, it will not tolerate being told to bear costs for them. (🧵)
We are hiring a Head of Private Office and two Private Secretaries at @AISecurityInst to work with me (Chief Strategy Officer) and @HZoete (Director).
If you’re sharp, ambitious, and want to help us run an important and fast-moving organisation, read on!
🚨🚨🚨 Job Alert with @CarnegieEndow@carnegienpp
We're hiring for a new Stanton fellow--pre-doc, post-doc, or junior faculty--to start next summer. Great opportunity to do policy-relevant nuclear research with a fun team.
Deadline: Nov 30. Link in next post.
Please repost!