Ben Korpan

[Ben Korpan @BenKorpan]

@DKokotajlo Plan S doesn't talk about the possibility of increasing human intelligence, which I think is what you'd actually want to do during a long shutdown.

[Jacques @JacquesThibs]

Roko’s Basilisk may be the most exceptional example of outsiders thinking some group (rationalists) take it extremely seriously, but in reality nobody in that group really thinks about it at all other than the occasional joke

[Eddie @chacaranda]

I dream of a nation where housing is so abundant landlords are competing for your rent application

[Richard Ngo @RichardMCNgo]

Notable exceptions: MIRI’s old agent foundations team, Steve Byrnes, Chris Olah, Davidad (though the latter three still partly fall into the trap).

Notable non-exceptions: Hinton, Bengio, Sutskever. At one point I expected that recruiting the very best ML researchers would help a lot (which is why I ran the original SF Alignment Workshop with Ilya).

But then they basically speedran the mistakes EA made in engaging with alignment: failing to deeply understand the problem, focusing on political games, founding a new lab, and the trap in my parent tweet.

That tweet was specifically inspired by Bengio’s new paper on Scientist AI. I’ve only skimmed it, but I’m pretty confident that it’s a central example of this trap. There are deep conceptual questions about the relationship between prediction and agency, what it means to have a goal, etc. This paper glosses over almost all of them in its attempt to have something it can pitch as a solution. See for instance its definition of a dangerous predictor below: there’s no insight there, just an attempt to get *something* you can prove *some* theorems about.

[Ben Korpan @BenKorpan]

> seeking to automate away all jobs, which is definitely something we should all support, but some people oppose because they don't understand economics.

I don't see how understanding of economics implies support. When better alternatives to horses for physical labor became available, the horses got sent to glue factories. This could go better than that, such as with the right policy decisions, but that doesn't follow from economics alone.

[Ben Korpan @BenKorpan]

@MunicipleOrrery @corsaren A return to a malthusian equilibrium would be horrible and worth coordinating to avoid.

[Ben Korpan @BenKorpan]

@corsaren The US also gives me a strong sense of "resting on its laurels" TBH. It can't even manage to repeal the Jones Act.

[Ben Korpan @BenKorpan]

@ilex_ulmus > The dirty little secret reason AI Safety people like talking about the libs’ denial is because it’s a topic on which they get to be aligned with the AI industry and other AI stans.

IDK, I think nerds just like being correct about things?

[Ben Korpan @BenKorpan]

> I predict that when you stop presenting AI Safety as requiring they kiss the tech culture ring, the libs will stop doing the mental gymnastics to deny dangerous capabilities.

What would be an example of "requiring they kiss the tech culture ring", and how would you present the ASI problem in a way less like that?

[Ben Korpan @BenKorpan]

Usually conclusions that seem absurd are in fact incorrect: otherwise they wouldn't seem absurd. This shouldn't prevent one from carefully making and examining arguments, since the sense of absurdness is far from totally reliable.

Also: in moral philosophy you have to keep the is-ought distinction in mind. A lot of bad arguments in moral philosophy are trying to derive ought from is. Moral intuitions are an important source of oughts! Where else are we to get our oughts from?

[Ben Korpan @BenKorpan]

@captgouda24 "The thing about someone shooting you tomorrow is that, if no one shoots you, you still die!"

[Ben Korpan @BenKorpan]

@captgouda24 @nabla_theta If a conclusion seems absurd then that is genuinely suggestive of the argument containing a flaw. In the case of rent control, the argument is in fact strong enough to survive this. The argument in your linked article, not so.

[Leo Gao @nabla_theta]

@captgouda24 modus tollens is absolutely a reasonable mode of argument. i also think there are big issues with your reasoning, but this is easier to point out.

[Ben Korpan @BenKorpan]

@captgouda24 So by that reasoning you're fine if someone shoots you today, because it's just "shuffling up a death that was already going to happen"? Also obviously the thing about humanity ceasing to exist if everyone dies at once.

[Jeremiah Johnson 🌐 @JeremiahDJohns]

One of the things I mourn the most in the current political era is that we've lost an important idea - that people who run for office should, at the very least, be competent people and good people. Not crazy. No major scandals, no outright evil views.

Nobody cares any more.

[Eliezer Yudkowsky @allTheYud]

REUTERS - "Sources" have revealed that the USG's decision to ban Mythos-class models was driven in part by their ominous name.

"We were in the Situation Room watching 'Resident Evil: Apocalypse' with the megacorp doing 'Project Nemesis'," said our source. "At one point Marco Rubio said 'Plot hole, why is any government allowing something called "Project Nemesis" to proceed? Would we really do that?' We all laughed for a few seconds and then we decided that nothing called Mythos was ever going to see the light of day."

Another source within the national security community confirmed that this sentiment was widespread. "If we let a corporation go forward with some project they call 'Mythos' and all hell broke loose, it'd be our own damn fault for lack of genre-savviness," said one highly-placed official. "Half of what we natsec guys do all day boils down to us desperately trying not to be the oblivious government from a stereotypical disaster movie."

PR representatives for Anthropic were slow to respond, but eventually got back to us.

"We're decently sure that Mythos 5 will not be the AI that destroys the world," said their statement. "Maybe 5.2 or 5.3? The name 'Mythos' refers to a large body of related fiction. It's not ominous at all. We definitely didn't get any secret kicks out of naming it that. This is unfair discrimination."

Asked whether the USG had similar plans to impose restrictions on GPT 5.5-Pro -- which some evaluations showed as having similarly quantified abilities to Mythos in some dimensions, but which did not produce the same reported sense of a new intelligence leap -- administration officials were ambiguous.

"Our decision to restrict AI models is a complex function of their evals, their felt intelligence, how ominously they're named, the size of bribe directed at the Trump administration, and how much that CEO has personally pissed off administration officials," said one highly placed source on background. "That's why we don't want any written laws about it. Say OpenAI starts delivering bribes on the same level that Nvidia uses to ignore export restrictions on B200s, and names their next model series Cutekitten. They could easily get up to GPT 6.2 before running into trouble. We're currently considering how to weigh those political realities against the prospect of GPT 6.2's internal deployment within OpenAI building another AI, that builds another AI, that destroys the entire Earth and creates an expanding wave of death in the form of self-replicating space probes."

Long-standing "AI safety" organizations declined to comment, saying that they were too torn between laughing and screaming to figure out which emotion should predominate.

[Ben Korpan @BenKorpan]

Ok that makes more sense, I misread your tweet as GPT-4 so I had 1.8T x 100 = 180T in my head. I don't think compute has got any cheaper since GPT-4, so if it is 10x GPT-4 then I'd expect it's only using ~1% of the weights on each token with MoE or whatever (IIUC GPT-4 already only used 1/8 of its weights on each token). My guess is that Mythos isn't actually that big, maybe GPT-4 size or smaller.

[Ben Korpan @BenKorpan]

@Moonicker @prerat 17.5T

[Nate Soares ⏹️ @So8res]

"The government will never do anything to hinder AI," they said.

Inaction yesterday does not imply inaction today. An export control directive came out of nowhere, and a ban on superintelligence could too. Inevitabilism is wrong.