The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Representatives Beyer, Moran, Whitesides and 20+ other members launching a new bipartisan AI Safety and Innovation Caucus.
All three are leaders on this issue and its great to see them emphasize that action on AI safety is too important for normal partisanship to block.
Does it feel like there's a new tech or AI crisis every day? Recent headlines are raising serious concerns about our digital safety and security.
Today, I'm joining @RepDonBeyer, @RepNateMoran, and over 20 of our colleagues to launch the bipartisan AI Safety and Innovation Caucus.
We're bringing Democrats and Republicans together to champion meaningful AI safety legislation, confront emerging threats, strengthen national security, all while advancing American innovation.
And we're just getting started. Stay tuned for more!
Wow - Senator Hickenlooper (CO) with a new letter calling on Senate leadership to make AI safety a top priority for the coming Congress, and to consider legislation "banning RSI" to stop "AI systems from autonomously advancing beyond our ability to control them."
Sen. Hickenlooper’s open letter to Sens. Schumer and Van Hollen says that the next majority leader should “consider banning Recursive Self-Improvement to prevent AI systems from autonomously advancing beyond our ability to control them.”
Seems like a good idea!
Sen. Hickenlooper’s open letter to Sens. Schumer and Van Hollen says that the next majority leader should “consider banning Recursive Self-Improvement to prevent AI systems from autonomously advancing beyond our ability to control them.”
Seems like a good idea!
It does seem like there is an emerging recognition that recursive self-improvement for AI is exceptionally dangerous, and that it makes sense for the government to step in and do something about it.
California’s new AI safety report is the most serious policy proposal for recursive self-improvement from any government I’ve seen. Very exciting!
It recommends the procedural machinery to actually slow down if & when we need to: verified compute disclosures for automated AI R&D, standby authority to cap that compute, emergency shutdown powers, mandatory loss-of-control reporting, and broadened whistleblower protections.
You can’t pace what you can’t see. If implemented, this would let the state see, and then act.
I hope the legislature moves on this when it’s back next month—AI developers seem on track to recursive self-improvement within 6-12 months, so we need these mechanisms practiced and working ASAP.
We scooped this morning that @SenatorHick is urging Schumer and Van Hollen to make exactly these AI priorities part of their leader races. Schumer has been working behind the scenes on AI generally and CVH is discussing in the context of Anthropic IPO
@kliu128 would love to see a version of the OECI plot with internal model release dates instead of public ones. for anthropic/metr, this seemed to matter significantly. sorry if i missed it!
ANTHROPIC: Recursive self-improvement "could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all." 😬
ALSO ANTHROPIC: We're excited to do recursive self-improvement. It is our key competitive advantage! 🙂
In the spring of 2023, Anthropic invited me to spend several weeks at their headquarters ahead of the launch of Claude 2, sitting in on meetings and interviewing dozens of employees to write the first major profile of the company in the NYT.
I just reread that story, and went back through my notebook in light of Anthropic's IPO. A few things that stuck out:
-- Anthropic was very small, very recently. When I visited, they had 160 employees in a small office on Sansome, a chatbot nobody had ever heard of, and seemed to have no interest in running a business. I wrote in my notebook that they seemed like "ambivalent capitalists"; their newly hired sales guy told me that the only reason he needed to make money at all was so that the safety researchers could save the world.
-- Things were not going well for them: their biggest shareholder (SBF) had been arrested for fraud, and OpenAI was eating their lunch with ChatGPT. (I later learned that they also had an expensive model training failure in 2023.) This was 3.5 years before they went public as one of the most valuable companies in the world, with 5,300 employees and 14x annual revenue growth. I guess they figured out the capitalism thing!
-- They were obviously and sincerely terrified of AI. Claude 2 was a very dumb model by today's standards, but every discussion I had there revolved around existential risk. Researchers told me they lost sleep over their fears of AI takeover, and made terrifying predictions of how soon AGI would arrive. It didn't seem fake or manufactured, but I didn't really get it! I wrote in my notebook: "I have never encountered a company that so badly needs a Klonopin." (I stand behind this, although I also need one now.)
-- Dario was in Italy when the NYT photographer wanted to take his picture, so his team rented an office in some random Italian town and staged it to look like his actual office, and they took the photos there. It had a rotary phone (?) and panic-room red lights, which I never really understood.
-- The Ants I talked to mostly struck me as sharp and thoughtful, although I did end up wondering whether they were too safety-obsessed to run an actual business. (I jotted down: "I do not know why you would invest in a company like this.") But I was struck by their earnestness, especially because I'd just spent several years covering social networks that didn't worry about the harms of their products at all.
Anyway, I'm feeling sort of conflicted about the Anthropic IPO. (As are a lot of people, including some at Anthropic!) I don't know if being public makes an AI company more or less likely to follow through on its safety commitments, and I know the company has changed a lot as it has grown. But on some level I'm also just like, damn, what a crazy story.
Here's the gift link, it's sort of fun to look back on the ancient runes of July 2023.
https://t.co/mqLMqElG63
Today is huge for the possibility of sane AI policy. Seriously, you should check out the report that the expert panel produced; it is *so* good.
As hopeful as I feel, I need to keep reminding myself that nothing in it is law yet, and there's a limited window to act. Feeling more optimistic than I have in a while, but boy is there a lot to be done, basically as soon as it possibly can.
My coworker @thlarsen makes a good point: While AI jaggedness is a real thing, modern AI is a lot less jagged than most people expected. A LOT less, unless I'm misremembering conversations a decade ago. Like, people used to expect a giant ecosystem of different AI services, rather than just a few flagship models that do everything.
Governor Gavin Newsom announced that GovOps, in consultation with Cal OES and four experts, has released the report "AI Safety in 2026 and Beyond."
It recommends mandatory baseline standards, independent verification, emergency shutdown capabilities, and the power to cap RSI. An important read for anyone working on the technical or policy side of frontier AI.
Very curious for more takes on how to reconcile this OpenAI information with the Anthropic METR report - is Anthropic just going way harder on automating AI research? (I think this is a bad thing tbc) Or if Anthropic reported the same metrics to those here would it be similar to what we see at OAI?
(Also happy that OpenAI released this info - they aren't required to and it has very real public benefit.)
We're sharing new data on how AI is changing research at OpenAI.
We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area.
This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems.
Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry.
https://t.co/xyGPcCkPCK
Okay--great honor to have been asked to serve on the expert panel with @alondra, @goldman and @robreich to advise @CAgovernor for a 2 month sprint on what to do about AI regulation. But the most incredible thing is to have been given the opportunity to help produce such a substantive and concrete report, chock-a-block with actionable steps to take NOW. A clear framework, state or federal. Accelerating the IVO ecosystem is central to everything, to rapidly build the capacity needed to MANDATE independent and ongoing risk assessments, reviews of safety frameworks for adequacy, and more. Revising critical incident reporting to ensure Hugging Face incidents must be disclosed. Requiring risk containment protocols (distributed "kill switch"). But the biggest news is around strong recommendations for an immediate priority on reporting on where companies are on the path to RSI (compute reporting, for example, with IVO verification). If there's one thing the legislature can get done before the end of the year, I hope it is this. https://t.co/UXhWQTWmbq
Great to see California taking the risks of recursive self-improving AI so seriously. This report lays out a vision for legislation that makes us safer, and I'd love to see California urgently turn it into law.
i am heartened to see such strong recommendations from the California government. they reflect the wisdom and expertise of the civil servants and independent experts who helped craft them.
@CAgovernor should act on these recommendations swiftly by calling for a special session.
five takeaways from the independent auditing section:
1. strong auditing requirements help build the capacity to administer them.
industry and safety skeptics often say that there are too few auditors to conduct frontier ai assessments, and therefore we shouldn’t require them.
as this piece recognizes, that logic is backwards — legal requirements create sustained demand for auditors! furthermore, they help channel existing state and independent capacity. the national labs, in particular, are well-equipped to support CBRN risk assessments.
2. frontier ai oversight should be dynamic and innovative.
i especially appreciate the strong, ambitious vision of what independent auditors could be — not just compliance box-tickers, but innovators and researchers.
different risks call for different specialists; mandatory audits will incentivize the creation of new risk assessment methods.
3. the public should have a window into auditor findings.
i welcome the call for the state to receive full reports on auditor findings, and for the public to see redacted reports (with redactions subject to government scrutiny.)
there are legitimate natsec and trade secret reasons to not share certain findings, but public reports are critical to supporting the broader research community.
4. the state must be empowered to act on auditor findings.
critically, the authors do not conceive of auditing as mere information gathering — it’s backed with state enforcement authority.
they specifically note that the state should be authorized to act on findings by ordering a pause on deployment, or by requiring risk mitigations.
5. strong independence standards are critical to ensuring audit integrity.
the report asks legislators to consider a couple different mechanisms to ensure auditor independence, including: government selection of auditors, auditor rotation, and fees on the labs to fund auditors.
i agree that auditor payment and selection is a critical policy design question, and look forward to seeing how the legislature approaches this.
Important work on the status of recursive self-improvement at OpenAI. As emphasized here, companies should be required to track and publicly report progress towards RSI to inform discussion of risks and to solve the technical and governance problems of human control.
@max_spero_@tracewoodgrains fwiw I think that anthropic and openai have different vibes about handoff (anthropic seems more openly excited/open to it comms wise) but are both are still just racing to handoff afaict (or at least this is what I currently think based on the evidence available)
Governor Newsom’s report on AI policy takes the challenge of the moment seriously by calling for:
- Mandatory embedded evaluators.
- State authority to issue orders in emergencies and to require corrective action.
- State authority to create mandatory minimum safety standards.
- Fixing scoping in existing state laws so the most risky frontier models and developers are covered.
- Pacing the frontier and limiting recursive self-improvement by limiting computational resources that can be used for automated AI R&D
Overall, this is a very strong report. Governor Newsom and the California legislature need to act now to put these recommendations into law.