As AI continues to progress, potentially indefinitely, does that mean we're coming to the end of the end of history? @FukuyamaFrancis and @JonKBateman explain, on The World Unpacked.
Watch their full conversation here: https://t.co/ECgJlrg5lr
@jachiam0 “Nothing ever happens” is a good Bayesian view on Iranian cyber attacks — or indeed any states’ cyber attacks — causing large scale real world disruption in the United States.
This was a profoundly impressive lift by a brilliant team: here's hoping *all* their recommendations are acted on, and set the pace for AI governance going into the new year.
This report is excellent, super useful and sharp and tightly focused on the most important things. Great job to everyone involved. I did not expect to see "by the way we're BANNING RECURSIVE SELF-IMPROVEMENT" on page six, but I'm here for it.
The IVO section is very good (a couple of key design questions, like who pays and selects, are explicitly not addressed/left to the legislature, but that's an encouraging sign IMO). I was expecting the "kill switch" thing to be kind of silly / not useful, but I was pleasantly surprised; they actually say some very useful things about how emergency shutdown authorities should work. The incident reporting seems great on a quick first read, very close to what I would've recommended (although for incident reporting a lot depends on leg-text-level details).
Surely that would be the least of our problems… but I do think “thousands of years” is an important reframing. The safety debate is often fought over timelines, yet if instead we just consider an arbitrarily long timeline, then the problem seems to be much much harder: *indefinitely* maintaining human primacy over arbitrarily smart super intelligence(s)…
You would think "powerful companies made something that might kill us all" would be a natural position for leftists to entertain but it turns out "I'm smarter than everyone in tech and in principle they can never do anything interesting" is a more central belief for many of them
Policymakers might try to slow down AI progress very soon. But most 'pacing' ideas would backfire: they're either too complex to work fast, or too much of a band-aid to last.
Today's essay looks at the politics of our least bad options to slow down AI.
https://t.co/kkeOBjvKYT
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
What happens when the end of history runs into the age of AI?
@FukuyamaFrancis joins @JonKBateman on @CarnegieEndow to discuss how technology may impact society, economies, and humans. https://t.co/WXUFfotIJO
This is an important essay. Important to label it as a consensus. Important to argue that humans *should* remain in control ("human chauvinism"). Important for the practical ideas of how to do so.
But is the silent consensus actually wrong in its prediction of the future? That's the heart of the matter, and here the essay says only this:
"Now, I happen to believe that the adherents of the silent consensus are wrong about what machine superintelligence will be. That they are missing something important about how the world works, that they conflate intelligence with power. If you described the capabilities of even today’s AI systems to an adherent of the silent consensus five or ten years ago, they would probably have predicted that joblessness would be much higher now than it in fact is, and that human disempowerment writ large would be far further along than it actually is. I believe that human collective intelligence and capability is far greater than the adherents of the silent consensus suppose, and that our societies will prove more robust than they predict.
But I could be wrong. [...]"
Thin gruel, in my view.
Really interesting work here by Vals evaluating frontier AI models ability to conduct drone attacks and properly identify targets.
These models are still too big (and far too slow) to do these tasks in the real world. But I forecast that today's frontier-level capabilities will be able to fit onboard large drones within about a year and on small drones within three years.
Any mitigations the frontier labs are taking on this might be largely irrelevant, because the models that will be fielded are those that are small enough to fit on these platforms, and the ones that militaries can freely modify -- ie. small open-weight models, which are today dominated by China.
Long thread here on my new paper and its findings:
https://t.co/wTihtjVPDN
@TheZvi@paulfchristiano He seems to be saying pause improvements to algo efficiency. Could still scale compute, refine data sets and RL environments etc?
It's so crazy that Elon, who generally seems to think of himself as a Great Man of History, has indeed led companies that have pulled entire industries years forward and dismantled a federal agency in a week, and is the literal largest donor in American politics, has such a sense of helplessness about this.
Call the president and explain that we're on track to soon have vastly superhuman AIs that we don't know how to control! We could ban RSI before Thanksgiving!
As AI continues to progress, potentially indefinitely, does that mean we're coming to the end of the end of history? @FukuyamaFrancis and @JonKBateman explain, on The World Unpacked.
Watch their full conversation here: https://t.co/ECgJlrg5lr
The White House pulls Dr. Heidi Overton’s nomination to lead the FDA amid growing concerns she didn’t have the votes to win Senate approval. https://t.co/CAtmQCO9Vh
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Software is becoming less reliable and secure. It's also becoming more plentiful, flexible, and capable. I don't think "Software Beirut" captures this. More like Jack London's Yukon -- where some make quick fortunes, others die in the snow.
I haven’t lived here in awhile, but I see no other way to answer the hundreds of texts/DMs asking me if the world is, indeed, ending.
In the span of six days, we’ve experienced;
11/12 — Microsoft outage
11/13 — Nationwide FAA ground stop
11/16 — Major US banking disruptions
11/17 — NSA’s classified internal network goes down
We need far more analysis of why so much is breaking at once. I don’t want to get out over my skis here, and what I am hearing does not contradict the messaging that most of this is the result of screwy updates.
But stepping back, the dots are starting to form a collective picture. We are seeing:
• An explosion in severe and critical vulnerabilities, with AI accelerating discovery to 3X the number of vulnerabilities discovered in the same period last year. This is collectively being referred to as “vulnageddon” or “the vulnpocalypse.”
• Mean time to exploit those vulnerabilities has collapsed to NEGATIVE SEVEN DAYS. AI allows attackers to find/exploit vulns far faster than software makers have time to fix them.
• Powerful AI-enabled offensive tools like ARTEX have been thrown into the hands of every third tier nation state, cybercriminal and digital arsonist, which logically creates havoc.
• This creates enormous pressure to compress patching windows and accelerate software updates, increasing the likelihood these fixes break things in unexpected, cascading ways.
• Pressure to just “AI everything” means we are rolling out automated AI tooling on top of brittle infrastructure that can’t support it.
My generation grew up in an extraordinary period of technological stability. We worked our way up the software stack, abstracting away layer after layer, until we lost a fundamental understanding of how the underlying systems actually work.
We were told to “move fast and break things.” VCS told us to celebrate the idea that “software would eat the world.” Now we’re telling ourselves we can simply “roll out AI” across everything, as though nothing underneath will buckle.
We never stopped to ask: SHOULD WE?
Should we be digitizing every last nook and cranny of our companies, infrastructure, healthcare, homes, cars, bodies, lives? Especially when the complexity, interdependence, and fragility we were introducing overwhelmed our ability to understand/secure/maintain the systems we built?
Nobody every asked: What if, in the name of progress, we were creating the ripest attack surface in human history?
We are now entirely dependent on software systems that almost nobody understands end to end, that increasingly few people can repair, and that attackers can now probe at machine speed. We are hearing that incident response firms, the 911 for breach victims, are backlogged six months. Even those paid on retainer lack the bandwidth to triage every incident. The cavalry is disappearing.
We are entering a period of unprecedented technological instability.
“Software Beirut” is an incredibly apt analogy. We are in for a period of cascading failures, whether triggered by attackers, bad updates, or our own accumulated complexity.
This, I’m sorry to say, is just the opening chapter.
Incredible news to wake up to: Obsolete is one of the Financial Times' best economics books of 2026!
"What is astonishing, as this book makes clear, is how complacent we are about the multiplicity of risks that this creates."
Writing a book with a strong perspective, there was always a risk Obsolete would be categorized as a polemic and discounted as such. But I aimed to make my argument with as much rigor and evidence as possible, and it's really nice to see it pay off!
https://t.co/u7NIfKJu5g
https://t.co/PM1lq0AWK7