Everett Smith

Researcher, controlling agents
Prev. COO, Watertight AI. Fellow at RAND. Georgetown IR.

[Everett Smith @DefenseBased]

What in the Jesus Christ is this

[Alex Bores @AlexBores]

Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI.

This is called Recursive Self Improvement (RSI).

OpenAI says they are on track to do so by March 2028.

Anthropic employees say they are on track to do so by 2027.

The METR report below suggests they are likely right.

This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.

We don't get another election before AI companies reach this threshold.

Congress should ban RSI, this year if possible.

This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)

The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.

Congress must ban RSI.

[Marcus Williams @Marcus_J_W]

🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.

To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

[Yo Shavit @yonashav]

Today many of the remaining decision-makers of the singularity get chosen, by the will of a people rather than by market competition. This may be the closest to an informed, broad-based, and thus morally legitimate decision that humanity gets. That is repulsively little. And yet, that we decide this way at all is glorious. Vote!!!

[Everett Smith @DefenseBased]

@LinchZhang Can you give some examples

[Everett Smith @DefenseBased]

Opus 5.5 is the first model that has scared a significant number of my smart generalist friends, a la "oh, the bell is tolling for us too"

[Everett Smith @DefenseBased]

@stanislavfort if the AI labs are able to sell outcomes instead of tokens, then maybe they'll be able to do the equivalent of charge you 30% less than what you would pay for a new phone or repair. That still depresses GDP but not as much.

[Everett Smith @DefenseBased]

@_conor_one_n @Anthropic This is awesome. The world is run by agentic people

[Everett Smith @DefenseBased]

@_conor_one_n How do you make your own practice exams + prep for the job?

[Everett Smith @DefenseBased]

Soon, the labs will have to decide whether to spend compute on "consumption" goods like curing cancer, aging, making Americans rich and ending global poverty or continue putting compute mostly towards the "capital" goods of robot factories and stronger AIs.

That's already happening, but the opportunity cost of compute, what we could do with it if invested in either consumption or capital goods, will go way up. The tradeoff will bite harder in both directions.

Maybe the labs will make some compute fund to cure the rare diseases of their employees and investors. I'd guess they will. Probably they'll do a couple high profile projects to eat the easy PR wins (see math papers).

But I expect the profit motive will encourage the labs to invest more in capital goods than would be socially optimal. Anthropic/OpenAI just don't capture most of the benefits of curing cancer, but they would bear most of the costs. That would be normal for a tech company, and normally they'd still capture enough of the surplus to make an amazing return, but the companies are racing hard, perhaps rationally hard but perhaps even irrationally hard. The competition for the frontier is zero sum, but they internalize all the costs + benefits of being #1. So much must be fed to the machine.

I think it will require courage and coordination to redirect resources towards the "consumption" goods that make life worth living. I hope that lab leaders and our government will have the virtue and skill to do so.

[Everett Smith @DefenseBased]

Broke: Racing through RSI to cure cancer
Bespoke: Reorienting away from RSI to cure cancer

[Everett Smith @DefenseBased]

Broke: Racing through RSI to cure cancer
Bespoke: Reorienting away from RSI to cure cancer

[Ben Goldhaber @BenGoldhaber]

This proposal from America First is excellent - re-orient the frontier towards curing cancer and not RSI!

[Everett Smith @DefenseBased]

I am very worried about the amount of power concentrating in tech corps, but there are ways for Anthropic + OpenAI to get around this.

And what’s the alternative for small countries? Many governments are deeply corrupt.

Getting stocks directly from a company seems safer than a UBI administered by a tribal kleptocracy.

[Everett Smith @DefenseBased]

Oh my god he’s right

[Herbie Bradley @herbiebradley]

@deepfates We're still so early

This is the least diluted autism will ever be

[Everett Smith @DefenseBased]

I am worried about the future concentration of media power and I hope AI can be used to break it up

[David Pfau @pfau]

So there's this thing called The Streisand Effect and this time it's about a new documentary. Click through, because if I say the name this tweet will probably get deboosted.

[Everett Smith @DefenseBased]

buy your girlfriend flowers, before you know what she actually thinks of them. Social morality is ex ante. You're judged for the predictable effects.

You should also normalize your predictions across people/time, unless you've got great specific data, otherwise it's too computationally expensive.

[Everett Smith @DefenseBased]

You are a good because you control how you treat other moral patients.

You must follow rules that define good behavior, that you can evaluate before knowing how the other person actually feels. Because you will never know.

It is right to

[Mikita Balesni @balesni]

Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.

I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.