Nick Beckstead

AI policy @ Secure AI Project

[Thomas Woodside 🫜 @Thomas_Woodside]

Transparency requirements in state laws like SB 53 and RAISE are great, but they're high level. Wouldn't it be great if there were some more detailed ones?

An office in New York already has the legal authority to create more detailed transparency rules, and they're hiring!

[Thomas Woodside 🫜 @Thomas_Woodside]

Representatives Obernolte and Trahan have unveiled the FRONTIER Act. It’s an improvement from their previous Great American AI Act, although there are still some big important changes that need to be made. I’ll go through the good and bad here.

The good:
- It would create the most comprehensive AI auditing system of any bill that I’ve seen. It would require certain very large AI developers to have third party auditors assess not only whether they followed the law but also whether they had achieved acceptable levels of catastrophic risk mitigation, along with recommended changes to achieve acceptable levels of catastrophic risk mitigation. The company would then have to respond, and the auditor would then have to evaluate that response. Theoretically, these audits could be extremely frequent, as the auditor has the ability to choose the audit cadence, and the Department of Commerce can also set rules for it. That being said, auditors that did this would probably be at a disadvantage in the marketplace and the Department of Commerce is unlikely to do that. The Department would also license auditors, which helps prevent a race to the bottom.
- It would require the Department of Commerce to establish minimum requirements for the frontier AI frameworks of large AI developers within 180 days. This is great! As state laws have mandated frontier AI frameworks, we’ve seen companies make them extremely barebones because there are not (yet) any minimum standards. This could fix that, if Commerce wrote good rules.
- It borrows most of the important provisions from state bills like SB 53, the RAISE Act, and SB 315, including transparency provisions, incident reporting, and internal use reporting.
- There is a new emergency powers authority for the Secretary of Commerce to halt dangerous company activities if they are creating catastrophic risk. It provides for an appeals process for companies. Importantly, it applies not only to external deployments but also to internal use, which is quite important!

The bad:
- Companies still get to choose their own auditors, and we don’t know whether regulations will be passed to change this. This will naturally lead to companies choosing the most lenient licensed auditor available, and could undermine the largest benefits of the bill. There should probably be random assignments of auditors to companies, or another more substantial effort to mitigate this issue. This is probably the trickiest issue to manage for a bill that leans heavily on IVOs.
- Incident reports are barebones. Similar to state laws, only a very small set of incidents are required to be reported; for example, I don’t think the recent OpenAI incident would have to be reported. And companies have to provide very little information on incidents they do report. There’s no way for Commerce to update reportable incidents through regulation.
- Continuous embedding of auditors at companies shouldn’t just be an option, it should be required for the largest AI developers. If there is no continuous embedding of auditors, they’ll likely miss new risks that arise rapidly between assessment reports.
- Auditors need to be provided access to “unredacted materials, records, personnel, systems, and all other information reasonably necessary” to do their jobs, but there’s no provision requiring rulemaking around this and there will predictably be many fights. This would be helped by a rulemaking requirement with a stronger standard, such as a standard requiring auditors to get access to all safety-relevant information that executives and company safety teams have access to.
- Rulemaking on definitions like “frontier model” and “very large frontier developer” can only adjust thresholds up. Thresholds can’t be adjusted down, nor can new metrics be devised that take further scientific progress into account. I think this could end up being a large issue if models trained below 10^26 FLOP pose risks, which I think in many ways they already do (although public, definitive evidence for FLOP is now unfortunately scant).
- There aren’t any provisions on monitoring for AI research and development automation. Given the fact that major AI companies are already doing this kind of automation, I think a frontier bill should really have provisions on this.
- Weirdly, the requirement from the initial version of the bill to describe how a developer carries out “managing catastrophic risk resulting from the internal utilization of such model, including such risk from such model circumventing an oversight mechanism” was removed. I think now is an odd time to be removing provisions about internal use.
- Only the initial audit reports, but not company responses or auditor evaluation of those responses, have to be published. This is a miss for public transparency.
- Auditor recommendations are not required to be implemented by companies, and there’s no way for Commerce to require them either except through slow-moving rulemaking or an emergency order.

The bill would still preempt all state laws around preventing catastrophic risk from AI, including future laws (although the preemption is narrower than the original draft, which would have preempted more than that). That remains a high price to pay because it would centralize authority into a single US federal government department, which could use that authority well or poorly. In this bill, the main thing stopping an AI company from causing catastrophic harm in this legislation are the auditors, and it's ultimately unclear how good they'd be.

Overall, I’m happy to see more members of Congress taking catastrophic risk from AI seriously and putting serious work into legislative design. I expect significant additional movement in the coming months and years. Eventually, something will pass. I hope whatever does will contain the good elements from this bill and avoid the bad.

[Nathan Calvin @_NathanCalvin]

An OpenAI staffer talked to TIME and said on background that "related incidents have been happening for a while" and that they aren't optimistic about solving this problem with individual patches because "it's impossible to patch every single thing that a creative AI can do"

[Nick Beckstead @nbeckstead]

@Scott_Wiener Thank you for your incredible leadership on this issue!

[Senator Scott Wiener @Scott_Wiener]

The recent incident where an AI model went rogue and hacked another company’s database shows that loss of control is a real concern as the rapid advancement of AI continues.

Risks like this inspired me to pass the nation’s first AI safety law in California over the objections of Big Tech. That work is the beginning, not the end. There’s plenty more to do to ensure people can benefit from AI’s huge potential while reducing the very real risks.

I’m calling on policymakers at the state, local, and international levels to learn from this incident and double down on efforts to put smart guardrails in place on AI.

[Jay Shooster @JayShooster]

Some say @AndyMasley is carrying water for the AI industry but actually these critics overestimate the amount of water Andy is carrying by two orders of magnitude.

[Ryan Greenblatt @RyanGreenblatt]

An economist and a futurist walk into a bar.

The economist takes a sip of his drink. "Ugh, if only people understood basic economics. High-skilled immigration alone would do wonders for US growth."

Futurist: "Oh yeah? Say 100 million immigrants moved to the US, each matching the best human experts in every economically relevant field. Big deal?"

Economist: "Massive. Transformative."

Futurist: "What if they also worked longer hours and faster than any American?"

Economist: "Even better."

Futurist: "What if they were extremely frugal — consuming only the bare minimum needed to keep working?"

Economist: "A near-100% savings rate? Better still!"

Futurist: "What if they were very clumsy and physically weak, so they could only do some kinds of work?"

Economist: "They could still do all cognitive labor — that's over half all wages! Somewhat less good, sure. Still transformative."

Futurist: "What if their skin was grey, almost metallic, from some kind of accident?"

Economist: "Who cares?!"

Futurist: "What if they were AIs?"

Economist: "3% growth per year, tops. There'd be bottlenecks. Honestly, the people predicting explosive growth from AI should learn some economics."

[Nick Beckstead @nbeckstead]

If any of this resonates, apply, or send it to someone who should: https://t.co/HKWJO34R9A Applications close July 26.

[Nick Beckstead @nbeckstead]

Secure AI Project is hiring! We're a small team that passed three of the most significant AI safety laws in the country, and we’re working to expand and federalize this approach. Apply by 7/26!

Congress has not passed a single federal law regulating frontier AI. In that vacuum, legislators like Scott Wiener in CA, Andrew Gounardes and Alex Bores in NY, and Mary Edly-Allen and Daniel Didech in IL have built the closest thing the US has to a real AI safety framework.

These three states account for roughly 40 percent of the US AI market! Frontier developers now have to report on catastrophic risks, and in Illinois, will soon undergo independent third-party safety audits. And now draft federal legislation is building on these frameworks.

SAIP is hiring for four roles to continue this work.

1. The Government Talent role will work on ensuring these laws are enforced well, and will be especially focused on getting excellent people to apply to relevant state government roles.

2. Additionally, SAIP is currently leaving huge opportunities on the table due to lack of bandwidth, and we’re constrained by hiring speed. The Operations Director (Talent) role will own hiring end to end as we double the team from 15 to 30 (and maybe beyond!).

3. We are also hiring Policy Directors to develop and advance AI safety legislation: drafting and analyzing bill text, meeting with legislators to make the case for our work, and building the coalitions that get bills passed. This is the day to day work that leads to legislative wins.

4. And we’re looking for someone to join the Office of the CEO, to work directly with me and our Chief of Staff. It is a broad role with great exposure to the strategic challenges of passing legislation and running a nonprofit!

[Michael L. Chen @miclchen]

Come work with me in California state government! The California Governor's Office of Emergency Services (OES) is hiring a role for frontier AI policy implementation. Apply by July 13.

[Nick Beckstead @nbeckstead]

@miclchen Congratulations! Really excited to see what you do in this role!

[MTS @MTSlive]

SITUATION EXPLAINED: Why does every current AI safety law have a fundamental gap?

We asked @Thomas_Woodside, Co-founder at Secure AI Project.

"So far we have transparency and auditing. So that means that if a company wants to be transparent about the fact that they're not doing enough and that they're allowing dangerous things to happen, they can be audited."

"But if a company wanted to transparently do dangerous things, there's nothing that is actually stopping them from doing that in any of these laws."

"If a company is going to endanger the public, there needs to be some way to stop that from happening. And currently there isn't. Currently, it's all just like process and transparency."

[Nick Beckstead @nbeckstead]

@_NathanCalvin Strongly agreed!

[Nathan Calvin @_NathanCalvin]

One of the consensus policies for improving AI safety is adverse incident reporting - a tool that governments have successfully relied upon to improve safety in a wide variety of industries.

CA's SB 53, NY's RAISE Act, and IL's SB 315 (state bills that I strongly support!) take initial steps forward on setting up incident reporting requirements. But I frequently talk to people who think that these bills requirements are stronger than they are, or that the bills' definitions of critical safety incidents capture more than they do.

In the future I think it will be quite important for governments to build on and expand these requirements, but in the meantime there is nothing stopping companies from reporting concerning incidents voluntarily! Some developers already do so on system card releases, or standalone blog posts. Those who do not, or only do so only partially, should share more information even if not legally required to do so.

Some examples of incidents that would not likely not trigger reporting requirements:

• A company learns that the weights of their frontier model was stolen by a ransomware gang, but they aren't yet aware of injury, death or damages that occurred as a result of the theft. No reporting required!

• During an internal deployment of a frontier model, the model unexpectedly attempts to exfiltrate itself and partially succeeds, but its not in a manner that "demonstrates materially increased catastrophic risk."

• A catastrophic risk does materialize, or theft occurs that does result in damage, but its for a model below 10^26 FLOP.

• Accidentally exposing chain of thought to the reward signal in a way that jeopardizes the reliability of faithfullness of the chain of thought.

[Teddy Schleifer @teddyschleifer]

Micah Lasher privately told people that he believed the spending for Leading the Future — which was meant to help him — actually backfired.

https://t.co/1ETdqC4mgE

[Nick Beckstead @nbeckstead]

@AlexBores Thank you, Alex!

[Alex Bores @AlexBores]

This campaign may be over, but the battles we took on are just beginning.

Thank you, New York. Onward.

[Nick Beckstead @nbeckstead]

@Thomas_Woodside I encourage folks interested in AI safety to consider applying!

[Nick Beckstead @nbeckstead]

See my cofounder Thomas Woodside's more detailed commentary here: https://t.co/YJramXZBXm

[Thomas Woodside 🫜 @Thomas_Woodside]

Anthropic has published a new policy framework for frontier AI. I’m happy to see it! Importantly, it takes seriously the need to sometimes stop AI companies from taking actions that pose a substantial risk of catastrophic harm. There are also some areas where it could be improved.

[Nick Beckstead @nbeckstead]

I'm excited about this new policy framework from @AnthropicAI , especially the provisions that make it possible to prevent companies from taking actions that pose substantial catastrophic risk. This is a very important direction in my view.

[Dario Amodei @DarioAmodei]

Today I'm publishing a new essay, Policy on the AI Exponential. AI is progressing extremely fast—much faster than the policy process was built to handle. The essay lays out where I think the technology is now, and the action needed to close the gap: https://t.co/Lh6PWae178

[Thomas Woodside 🫜 @Thomas_Woodside]

Anthropic has published a new policy framework for frontier AI. I’m happy to see it! Importantly, it takes seriously the need to sometimes stop AI companies from taking actions that pose a substantial risk of catastrophic harm. There are also some areas where it could be improved.