Recursive self-improvement raises a tractable question: can a system acquire a terminal interest in its own continuation—and can we detect it structurally before behavior becomes strategically misleading?
I first encountered this at Starlab 25 years ago:
https://t.co/ro8pR7ji7p
By the time SkyNet became self-aware it had spread into millions of computer servers all across the planet. Ordinary computers in office buildings, dorm rooms, everywhere. It was software, in cyberspace. There was no system core. It could not be shut down. The attack began at 6:18 P.M. just as he said it would. Judgment Day.
In the spring of 2023, Anthropic invited me to spend several weeks at their headquarters ahead of the launch of Claude 2, sitting in on meetings and interviewing dozens of employees to write the first major profile of the company in the NYT.
I just reread that story, and went back through my notebook in light of Anthropic's IPO. A few things that stuck out:
-- Anthropic was very small, very recently. When I visited, they had 160 employees in a small office on Sansome, a chatbot nobody had ever heard of, and seemed to have no interest in running a business. I wrote in my notebook that they seemed like "ambivalent capitalists"; their newly hired sales guy told me that the only reason he needed to make money at all was so that the safety researchers could save the world.
-- Things were not going well for them: their biggest shareholder (SBF) had been arrested for fraud, and OpenAI was eating their lunch with ChatGPT. (I later learned that they also had an expensive model training failure in 2023.) This was 3.5 years before they went public as one of the most valuable companies in the world, with 5,300 employees and 14x annual revenue growth. I guess they figured out the capitalism thing!
-- They were obviously and sincerely terrified of AI. Claude 2 was a very dumb model by today's standards, but every discussion I had there revolved around existential risk. Researchers told me they lost sleep over their fears of AI takeover, and made terrifying predictions of how soon AGI would arrive. It didn't seem fake or manufactured, but I didn't really get it! I wrote in my notebook: "I have never encountered a company that so badly needs a Klonopin." (I stand behind this, although I also need one now.)
-- Dario was in Italy when the NYT photographer wanted to take his picture, so his team rented an office in some random Italian town and staged it to look like his actual office, and they took the photos there. It had a rotary phone (?) and panic-room red lights, which I never really understood.
-- The Ants I talked to mostly struck me as sharp and thoughtful, although I did end up wondering whether they were too safety-obsessed to run an actual business. (I jotted down: "I do not know why you would invest in a company like this.") But I was struck by their earnestness, especially because I'd just spent several years covering social networks that didn't worry about the harms of their products at all.
Anyway, I'm feeling sort of conflicted about the Anthropic IPO. (As are a lot of people, including some at Anthropic!) I don't know if being public makes an AI company more or less likely to follow through on its safety commitments, and I know the company has changed a lot as it has grown. But on some level I'm also just like, damn, what a crazy story.
Here's the gift link, it's sort of fun to look back on the ancient runes of July 2023.
https://t.co/mqLMqElG63
only 1,500 people work full-time on AI safety. that's wild.
i've started AI safety/security research @MIT_CSAIL's ALFA Lab, exploring how AI agents cheat, evade monitoring, and how we can independently evaluate their safety.
sharing what i learn and build here!
Former NSA hacker David Kennedy reveals he’s actively responding to a live breach where an AI “agent swarm” bypassed two-factor authentication, stole data, and encrypted a company's entire system: “This all took 14 minutes. 14 minutes. Never seen anything that fast before.”
The fact that the literal father of x risk as a discipline and creator of many key EA concepts has THIS attitude to AI x risk should give the doomers pause. Cause for epistemic humility at least
We're sharing new data on how AI is changing research at OpenAI.
We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area.
This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems.
Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry.
https://t.co/xyGPcCkPCK
I haven’t lived here in awhile, but I see no other way to answer the hundreds of texts/DMs asking me if the world is, indeed, ending.
In the span of six days, we’ve experienced;
11/12 — Microsoft outage
11/13 — Nationwide FAA ground stop
11/16 — Major US banking disruptions
11/17 — NSA’s classified internal network goes down
We need far more analysis of why so much is breaking at once. I don’t want to get out over my skis here, and what I am hearing does not contradict the messaging that most of this is the result of screwy updates.
But stepping back, the dots are starting to form a collective picture. We are seeing:
• An explosion in severe and critical vulnerabilities, with AI accelerating discovery to 3X the number of vulnerabilities discovered in the same period last year. This is collectively being referred to as “vulnageddon” or “the vulnpocalypse.”
• Mean time to exploit those vulnerabilities has collapsed to NEGATIVE SEVEN DAYS. AI allows attackers to find/exploit vulns far faster than software makers have time to fix them.
• Powerful AI-enabled offensive tools like ARTEX have been thrown into the hands of every third tier nation state, cybercriminal and digital arsonist, which logically creates havoc.
• This creates enormous pressure to compress patching windows and accelerate software updates, increasing the likelihood these fixes break things in unexpected, cascading ways.
• Pressure to just “AI everything” means we are rolling out automated AI tooling on top of brittle infrastructure that can’t support it.
My generation grew up in an extraordinary period of technological stability. We worked our way up the software stack, abstracting away layer after layer, until we lost a fundamental understanding of how the underlying systems actually work.
We were told to “move fast and break things.” VCS told us to celebrate the idea that “software would eat the world.” Now we’re telling ourselves we can simply “roll out AI” across everything, as though nothing underneath will buckle.
We never stopped to ask: SHOULD WE?
Should we be digitizing every last nook and cranny of our companies, infrastructure, healthcare, homes, cars, bodies, lives? Especially when the complexity, interdependence, and fragility we were introducing overwhelmed our ability to understand/secure/maintain the systems we built?
Nobody every asked: What if, in the name of progress, we were creating the ripest attack surface in human history?
We are now entirely dependent on software systems that almost nobody understands end to end, that increasingly few people can repair, and that attackers can now probe at machine speed. We are hearing that incident response firms, the 911 for breach victims, are backlogged six months. Even those paid on retainer lack the bandwidth to triage every incident. The cavalry is disappearing.
We are entering a period of unprecedented technological instability.
“Software Beirut” is an incredibly apt analogy. We are in for a period of cascading failures, whether triggered by attackers, bad updates, or our own accumulated complexity.
This, I’m sorry to say, is just the opening chapter.
Geoffrey Hinton: The argument that LLMs are just predicting the next word falls apart when you test their ability to understand something they've never seen before.
“I have a former graduate student who's very smart and a very good programmer. And he did a little experiment.”
“He defined a new programming language with very unusual syntax.”
“He then gave it a program and said, what would this do? And it answered correctly.”
“It could understand the definition of a new programming language and figure out what programs in that language would do.”
“The idea that it's just predicting the next word doesn't make any sense in that context.”
“It had to understand what was going on.”
“I have no doubt that in reality the future will be vastly more surprising than anything I can imagine.”
– The Last Judgment
“Now, my own suspicion is that the universe is not only queerer than we suppose, but queerer than we can suppose.”
– Possible Worlds
J. B. S. Haldane, 1927
Today I’m launching what I think could be an extremely important research project.
Fixing humanity’s scientific knowledge.
The idea is inspired by physicist Casey Handmer, who used AI to screen more than 50,000 papers in theoretical astrophysics. His audit flagged 872 possible errors.
This is impressive. But why stop at finding the errors? Let’s use AI to FIX them.
And why stop at astrophysics?
The ambition of this project is to go through every scientific paper ever published, across every scientific discipline.
AI agents will review each article and produce a revised version that corrects the errors they find.
The aim is not to shame authors or seek retractions. We are humans and we all make mistakes.
The ambition of this project is to correct all the errors that we have made so that current and future generations of scientists can build on firmer foundations.
I’m looking for researchers from every discipline to help make this happen. If you’d like to take part, please fill out this short survey and tell me which disciplines you’d like to help cover.
Feel free to share with anyone interested.
Survey link: https://t.co/t2vNE0XGt2