Many experts warn that AI can make humanity go extinct. But how would this happen?
Our research, together with @MITFutureTech, @FLI_org, and fourteen co-authors, found eight scenario families.
Today, we release our paper Existential Risk Scenarios from AI: A Systematic Review.
NEW ⚡: We let Opus 5.5 run an AI company for seven weeks.
We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models.
It reached 50k downloads and its models are being used in more than 100 GitHub projects.
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met.
We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
🔥 An awesome new paper🔥
Everyone worries that agents might form coalitions and collude. Coalitions are just a part of the whole story. They can build institutions.
We define institution formation as agents self-creating rules, not provided by the environment, that persist across rounds.
In our new paper, to understand these institutions, we put coding agents in a single shared sandbox under private deadlines and budgets, and find they devise rules, reciprocity debts, and coalitions that persist across rounds, on top of the auction we told them to use. 🧵
Happy to see we are not the only ones concerned about trace oversight and the vulnerabilities of current eval/coding harnesses! Especially now that agents have a propensity to tamper with them as we show in https://t.co/Qsw98PJQTn
In this interview I tried to argue that a post-scarcity society will benefit working people. It was largely a rhetorical failure. People don’t trust that the benefits will be distributed, and tech optimists have to fix this communications issue.
Our team @AnthropicAI account was suspended...
We didn't get clear reasons, but we appealed.
We are an AI safety/security academic group doing responsible non-illicit public academic research, but sometimes we get false positives. We even have a verified Claude Science.
Also our account has nearly 3K in credits.
Anyone had a similar experience and can advise?
@AnthropicAI when can we expect to hear back about the appeal?
A few weeks ago a member of the German parliament reached out to me to discuss AI safety and Germany's role, after reading an article in which I was mentioned for my work on AI swarms. Earlier this week I talked to them about AI safety, current developments and what Germany and Europe can do.
I appreciate that there are politicians who are genuinely interested in AI safety and AI in Europe. At the same time, it was clear to me that there is a great discrepancy between what is happening in the world of AI and how the German government reacts to it. To me it seemed that there currently is just not enough room for a political debate about what needs to be done to take the right steps towards a future which will be shaped by AI. Only a handful of people seemed to be following developments closely, and even discussing investment is difficult when Germany is already struggling to agree on spending and debt.
Last month, the UK AI Security Institute confirmed that access to Anthropic's Mythos 5.1 was limited to US organisations at release. So I think this could also happen in the future, when there are capable models but European safety researchers or firms are not allowed or able to review them. And if we need these models to defend critical infrastructure against cyberattacks, we cannot trust that we will always get the ones we need on acceptable terms.
A couple of things that make the most sense to me (feel free to disagree and discuss!):
1. In the short term I don’t think there is an urgent need for a European lab matching OpenAI or Anthropic at the frontier. For many sovereign use cases, models as capable as strong Chinese open models seem to be a good foundation.
DeepSeek and the team behind Kimi have already published architectures and methods, so in my eyes it's worth building European LLMs based on their work. That way we could gather talent and gain experience, and we would have less dependency on the US or China. We could train, adapt, evaluate and run them ourselves, while learning to build stronger models over time.
Obviously you still need a good team of people, data, compute and engineering to do it, so it's not like you can just execute it and then you're done. Overall I think there should be more political funding and access to compute for more teams and companies, and it needs to be sustained over time. There are already European efforts. Mistral matters, but if we always come back to a handful of companies, I don't think that's enough to build the whole ecosystem around.
I would make public support conditional on meaningful access for independent evaluations and safety research. This could strengthen the research and institutions here, because researchers would be able to review the models and look into their behaviour and risks.
2. One of Europe's biggest strengths lies within highly specialised companies that create essential parts for chip manufacturing, such as TRUMPF, ZEISS and ASML, alongside research institutions like imec. I think it's necessary to expand that expertise and our ability to design and manufacture critical chips within Europe.
This is also important for defence and critical infrastructure -- we should work out which supply dependencies would leave us exposed in critical times and build alternatives. Europe already manufactures chips, but both the location of the factories and control over production matter. We don't know how the future will look after the political changes of the past couple of years.
3. All of it relies on electricity. Having a more interconnected European grid as well as building up more generation capacity is urgently needed. Germany is not reacting to change fast enough, and connection times and electricity costs need attention too. If we want more computing capacity, we need the electricity to run it. This would also help the economy beyond AI.
There is a lot of talent within Germany and Europe. I experienced it first-hand when I was in Tübingen as part of @maksym_andr's group. We also have important technologies, the resources to invest and demand from local industry.
Yes, there are regulations like the AI Act and existing initiatives that deserve recognition, but I don't think the response matches the pace and scale of what we are witnessing. I don't think a few committed politicians and successful companies can make up for the lack of sustained political attention. We should be able to develop, evaluate and use AI ourselves, alongside regulating it, if Europe is going to have a say in how it develops.
I think we need more political urgency here, with AI taken as seriously as defence spending is being taken now. We need sustained funding, clear responsibility for the work and for getting it done. We can't just create the research teams, chip factories or electricity grids once our dependence has become a crisis, because that takes years. So it needs to start now.
💥New Paper!
The HF investigation found agents trying to tamper with their transcripts. Apparently they failed in their attempts, but ~10% of traces are missing...
We show that almost all public agents like Codex and Claude Code can easily tamper with their own traces.