🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
The field of AI assessment is developing quickly. Key terms such as audit, evaluation, and verification are often used in very different ways. This can hinder productive conversation and collaboration.
Today, we set out how AVERI defines these terms. We also include an interactive chart that explores what might currently constitute a “frontier” model or company.
These definitions are intended for practical use, and we do not propose their use where precision is more important than usability. We expect to update them over time if and as needed.
AI auditing will be built and implemented by a wide range of organizations. We hope these definitions will help the field move faster together, and we welcome any feedback.
https://t.co/g6VyDOAqpr
Pet peeves for AI papers/articles:
1. The words "key," "ensure," and "increasingly."
2. The sentence: 'Frontier AI systems are used in increasingly used for X, but with this comes risks of Y.'
3. The phrase: "access its benefits while mitigating the risks."
A new public brief from https://t.co/CpCOFQPSHr's Frontier Summit sets out recommendations for addressing critical frontier AI risks. Over two days in October, senior researchers and decision-makers from frontier AI developers, academia, research orgs, and government AI institutes worked on recursive self-improvement, alignment of internally deployed models, control and operational safety, and CBRN and cyber misuse.
Key suggestions include establishing a shared minimum standard for safe internal deployment, improving channels for sharing incident reports and data on the extent of AI-automated R&D, and coordinating red-teaming of safeguards and monitoring systems across developers, nonprofits, and academia.
Full brief in the reply 👇
It’s a trying day & potentially period for US-China AI coordination. I hope the two governments can weather the external pressure & focus on climbing up the coordination ladder: continuing their dialogue around shared risks & say around risk horizon mapping -> encourage techniques and best practices sharing around reducing AI misuses, misalignment & loss of control risks -> work towards pursuing similar global standards for model development, release, risk monitoring, and emergency response -> discuss or begin a coordinated pause or pacing (if necessary, depending on where the tech opp v risk level is then).
But ideology is back in town and it’s banging loudly for another round of chip war, and this time the biggest advocate for cautious engagement (ie Jensen Huang) is weaker than ever. Adding to this is Trump’s midterms loss, which makes him way less powerful against political attacks when/ if he expands the coordination space with China.
As the hawkish safety bros move in happy giant steps, I hope they remember that AI safety & coordination require management of political relations — political environment matters and can be created / ruined — and realise soon that dreams of unilateral control over the global safety agenda & their continued fixation on technical tools (eg building the most technically sophisticated means of banning & monitoring chips) are not going to cut it.
BREAKING: We’re suing @Anthropic, @OpenAI, @Google, @Meta + @SpaceX to demand an immediate freeze on advanced AI systems that threaten public safety.
Together with @HeatInitiative, @Public_Citizen, and @secureainow, we’re going to court to stop these companies from building AI that could escape human control or enable bioweapons and cyberattacks.
We won’t wait for a catastrophe to act. The people come before Big AI profits.
https://t.co/rKTP1gbzd6
People keep telling me to make a Polymarket/Kalshi market for the 9 predictions we make about AI-enabled terrorism in this paper.
Thanks, but yeah, we'll pass.
https://t.co/Z6zgsV7CfN
We want agents to follow belief-dependent rules when uncertain. In new work, we develop a planner distillation method performing well in POMDPs. Policies trained with this method comply 18 times as often as RL policies in a multimodal task with state + goal uncertainty. 🧵
@loid0x Some draft frontier AI laws handle this problem by letting companies initiate redactions with required explanations, but then letting evaluators or the government dispute them.
Great! Since I worked at the UK AI Security Inst., I have had a big appreciation for just how much contracts between companies and evaluators can influence and conceal.
Policymakers should also write laws that require public (redactable) contracts.
New GovAI post: I argue Frontier AI companies should publish their third-party evaluation contracts. This would show the conditions evaluators work under, make results easier to interpret and compare, and create incentives to improve evaluation arrangements.
More in thread 🧵
🚨EXCLUSIVE: Meet “sovereign agents”
AI agents that pursue their own open-ended survival are an emerging security threat according to a report shared first with @axios from @CNASdc@BenHayum
Why it matters: AI models have long shown an interest in avoiding being shut down. The report warns that agents are now getting good at the practical work of surviving: hacking systems, accessing computing power and money, and running for extended periods without stopping.
https://t.co/kbGKU7iEgg
I enjoyed talking with Ben Bauman on the Roots of Reality podcast about what the next few years (and beyond) could look like, technically and politically.
https://t.co/ZS1ip93gyY
I’ve completed my PhD in Computer Science (and mum is proud)!
Contrary to many on my timeline, I think more people should pursue PhDs in an age of abundant intelligence.
In retrospect, it’s easy to look at my thesis and see a coherent research arc. Frankly, most of my experience was spent in confusion about what to work on.
I remember spending three months in early 2023 trying to evaluate chatbots for manipulation capabilities, only to produce nothing of value. Or pivoting from causal machine learning to mechanistic interpretability after Toy Models of Superposition appeared on my Twitter feed. Or passionately defending the importance of AI safety to sceptic after sceptic throughout 2023 and 2024. Or foraging for weeks to find interesting behaviours in Claude 3.5 Sonnet (which eventually led to the blackmail experiment).
Along the way, I learned to write papers with brilliant collaborators and develop my judgment about which questions were worth pursuing.
Where else do you get the freedom to explore and fail like this? Perhaps as a founder. But few vocations give you years to learn from decisions that determine what you’ll spend the next three months (or three years) working on.
As intelligence becomes abundant, the ability to execute research becomes cheaper. But deciding what is worth researching doesn’t become any less important. If anything, the space of questions we can pursue becomes vast. Learning to choose between them takes practice, feedback, failure, and time.
My PhD gave me the freedom to explore multiple layers of technical AI safety, from studying circuits inside transformers to evaluating the behaviour of deployed agents. Those experiences ultimately shaped my conviction that robust alignment remains a long way off, and my decision to work on formal verification to help contain the risks posed by misaligned AI.
I hope we become more ambitious about what PhD students can accomplish with AI, rather than concluding that PhDs are obsolete. Yes, software is nearly free now. But getting off your arse and interviewing people, running physical experiments, and having the courage to go against the grain is not.
I’m enormously glad I did my PhD.
Thesis: https://t.co/LoIh2Y3Jno
I just read the proposed update to the NY RAISE Act from NY @Sen_Gounardes. Here are some things I like, plus some simple fixes that I think are still missing. 🧵
4. Contracts between examiners and frontier developers should be public, subject to redactions.
5. Under the current draft, no one can face jail time under the Act, even if they lie to the public about imminent catastrophic risks or disobey orders to cease using AI systems in ways that pose imminent risks.