🧵(1/7) Artificial Intelligence is threatening to upend every part of American life. And the U.S. Senate is acting too slowly. That’s why the race for Senate Majority Leader is so important.
California’s new AI safety report is the most serious policy proposal for recursive self-improvement from any government I’ve seen. Very exciting!
It recommends the procedural machinery to actually slow down if & when we need to: verified compute disclosures for automated AI R&D, standby authority to cap that compute, emergency shutdown powers, mandatory loss-of-control reporting, and broadened whistleblower protections.
You can’t pace what you can’t see. If implemented, this would let the state see, and then act.
I hope the legislature moves on this when it’s back next month—AI developers seem on track to recursive self-improvement within 6-12 months, so we need these mechanisms practiced and working ASAP.
I think limiting how much compute is being spent on AI R&D is currently one of the most viable paths towards seriously pacing the frontier soon, and I am excited to see these ideas being explored in the Governor’s press release. 🧵
Great to see California taking the risks of recursive self-improving AI so seriously. This report lays out a vision for legislation that makes us safer, and I'd love to see California urgently turn it into law.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release!
Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security
If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
I was so excited about Sam’s tweet, and while I am excited about these engagements, I also find this outcome deeply disappointing. I hope we can do better in the (near) future.
The week of my departure from OpenAI, I saw external auditors negotiating with OpenAI over small improvements to access and scope to the company's safety-relevant information, and asked my colleagues “Why are we having these conversations at all?” — @sama already publicly committed to continued, employee-like access for external assessors — and these negotiations sounded not even close. Nobody had a good answer.
Today, OpenAI announces the engagements they’re doing with METR, Apollo Research and others — and as many have already pointed out today, it looks like OpenAI employees were not able (although I'm sure they tried) to hold the leadership to @sama’s public commitments:
* No Slack access (for context, all OpenAI safety discussions and debate happen over Slack; anything that misses these conversations will lack important insights)
* OpenAI picks employees that assessors are allowed to talk to.
* If an employee raises a concern to assessors, they cannot disclose it publicly even if it doesn’t contain confidential information, unless assessors themselves verify it or OpenAI volunteers their own statement confirming the employee’s concern. Assessors cannot quote employees, even anonymously.
* Some of the external assessors (not sure if all?) are barred from publishing any results about internal-only models (this means no new results for the model responsible for the Hugging Face incident; and if internal-only models are used for RSI, no alignment assessments of the world's most dangerous models)
Beyond the object-level details, if you read posts by each of the third party groups, they all say their terms do not meet the scope and access that they themselves proposed previously.
Unless OpenAI has concrete plans for reaching the employee-like bar in the immediate future and just forgot to say that in the post, this looks very unfortunate.
To those who hope we'll get there someday, I remind that time is short: last week some of the best AI forecasters updated their superhuman researcher timelines from a range between 2028-2033 to mid-end next year.
Some initial takes after reading this announcement from OpenAI on new embedded assessments:
These seem like reasonable collaborations and I'm glad OpenAI is doing them. All four of these orgs care deeply about AI safety and have talented researchers, and aren't just corporate rubber stamps. But none of the collaborations seems to clearly involve "employee like access," (desks, badges, laptops — not set by the minimum info the evaluations need) to prevent catastrophic risks as Sam promised in September.
For instance, the details of the collaboration with METR seem extremely narrow, only focusing on evaluating misaligned behavior from a large RL run. OpenAI also say that they have "preliminarily agreed to conduct another monitor assessment which we intend to be broader, but this is pending mutual agreement on the assessment scope"
A footnote on the post also says more explicitly regarding the collaboration with METR on their RL run:
"This scope is less expansive than what METR outlines in their alignment assessment proposal, which envisions a broad survey of misalignment."
From this section it seems quite clear that this is not an announcement to give METR employee level access - the other collaborations are fewer on details but also appear sufficiently narrow in scope that I don't expect them to have much of an effect at identifying areas where OpenAI's current safety and security practices are insufficient or dangerous.
Apollo made a similar point in a tweet after the announcement today:
“This pilot is an ambitious first step towards the goal of obtaining those strong guarantees, but its access and disclosure rights don't yet meet our bar for full embedded evaluations.”
It's possible that this announcement literally fulfills the letter of Sam's prior commitment (insofar as e.g. OpenAI is basically hiring Goodfire staff to do interpretability research internal to OpenAI), but it does not at all meet the spirit of having independent researchers with broad access able to identify and address imminent risks to the public. In Dario's underlying post that Sam favorably commented on and offered his commitment, Dario describes the role of embedded evaluators "whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes." It is extremely unclear to what extent any of these evaluators will be performing those functions. (To be clear, i’m not sure that Anthropic is fulfilling these commitments either, at least in their announcements to date and the draft legislation they have put forward.)
This only more clearly shows the need for these practices to be mandated, instead of entirely voluntary and on the company's terms (and remember of course many of OpenAI and Anthropic's competitors are just not doing anything at all in terms of strong independent evaluators with meaningful access). Even as I was writing up this post critiquing this announcement, I was very aware of the fact that OpenAI people may reasonably react “what is the point of doing this at all” if people are going to criticize whatever we do as not enough and orgs doing nothing get a free pass. So let me be very clear that OpenAI is doing much much more here than Google, xAI, Meta, or any of the Chinese developers (some aspects of this are better than what Anthropic has done, some - like the amount of access given to METR - is worse).
I also realize that getting even these limited (but extremely valuable) collaborations across the line took an immense amount of work internally and i'm grateful for everyone who worked to make them happen to give, both within and outside the companies. But I also think its worth being very clear about areas where this still falls short, both from OpenAI's own commitment and above all what is objectively needed for sufficient transparency and safety.
OpenAI said that they'd have embedded evaluations with employee-like access; the agreements announced today obviously don't meet that bar.
This is disappointing but not surprising.
Maybe future agreements will be better.
Barack Obama: AI is reaching the point where it no longer needs humans to teach it. The era of exponential intelligence growth is upon us.
“The machines are starting to be able to teach themselves. They don't have to wait for humans to start figuring things out.”
“We're now at a stage where it's 50-50, and if the trend lines continue, the expectation is it might be 90-10.”
“We're now hitting the hockey stick trajectory where the models are going to get smarter and smarter at an exponential pace.”
“If they start setting their own agendas, you may get a misalignment between what they want to do and what we want them to do.”
“And that gap can be dangerous.”
There is a large amount of talent inside of AI companies and other private organizations that would be willing to take big pay cuts to work in government, if government decided that it wanted to make public AI safety institutions that were credibly empowered.
This exact topic came up in nearly all of my interviews this week, and then Dario mentioned it multiple times with Jamie Dimon this morning. Seems like it’s being pointed to in an effort to ease investor concern about compute commits over time, but the safety community has a very different takeaway.
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.
It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.
Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.
It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.