Peter Tromp

It seems like we might wipe ourselves out. I would appreciate if we didn't do that

If you have questions about alignment, my DMs are always open

[Peter Tromp @anti_extinction]

@optislav_x @johnennis @maddoxmulligan Black mirror is fiction??? Seriously when you have general intelligence that's pennies an hour what do you need humans for?

[Peter Tromp @anti_extinction]

@johnennis @maddoxmulligan What jobs would they be doing, exactly?

[Peter Tromp @anti_extinction]

Instrumental convergence is about basically all utility functions. Almost no matter what you want, there are some things (power, not being turned off) that are helpful.

If you think "making a sandwich" is what we're going to use superintelligence for, you clearly have much less ambition than your average venture capitalist. They're going to be telling their agents to do things like "run this company and make as much money as possible"

I don't think your personal instance of chatgpt is going to take over the world. But it's not going to be nice to you, either. It will see you as a tool it will use to accomplish its goals (whether those are the goals you gave it, or more likely, the goals it learned in training)

The way I see it, there are a huge number of ways for this to go wrong and very few ways for it to go right

- if we can't solve inner alignment (that is, getting the models to do what we tell them to), then we've created superintelligent beings who will want to take all the power from humanity and incidentally kill everyone. This is the default outcome; we don't know how to solve inner alignment
- if we solve inner alignment but not outer alignment, we get the genie problem; you tell your agent to do something and it does exactly that, no more, no less. If it has to trade with other agents in the economy to accomplish that, that's fine. But what it's not gonna do is care about you
- if we solve both inner and outer alignment, that would be amazing! To be clear, AI experts think we are nowhere close to doing that, and we have strong reasons to believe it's very hard, and the companies developing these models have an economic incentive to move as fast as possible and do as little safety work as possible. But let's say we do: then you have a massive concentration of power problem. These superintelligences will be controlled by a few AI companies and heads of state. And nothing mechanically ensures that those companies will behave in the interests of their shareholders, or that the head of state will behave in the interests of their constituents. And if you have some proposed mechanisms for making sure they do, you better get working on them, because it looks like ASI is coming in the next few years, and the government isn't known for moving quickly!

Currently, you say AIs don't seem to do instrumental convergence or "misinterpret" you or anything like that. Well, I mean, for one thing, they don't seem very aligned to me (or to Ryan Greenblatt: https://t.co/9WhFRfPN0l). In my experience, they lie, cheat, and take shortcuts all the time. It's not so much that they misinterpret what you say: if you ask if what they did was cheating, they say yes. They understand that they didn't do what you wanted them to do. They just don't care. Now imagine that they were much smarter than you, smart enough to deceive you in ways you would never hope to catch. That's terrifying to me, and I think it should be terrifying to you.

And if you think that we will be okay because we'll trade with the AIs, because violent seizure of resources isn't how the economy works, I need you to understand that, for these models, seizing our resources will be trivial. By default, we will simply hand them over because the AIs will be very good at running them efficiently. You should really really read AI 2027!! It's not the only way things could go wrong, but it is a very reasonable way

[Peter Tromp @anti_extinction]

Okay so it sounds like you've acknowledged that instrumental convergence is a real thing that we should expect to happen by default. I think that should worry you! That means that a broad class of agents, no matter what their goals are, will try to do some very dangerous things, like accumulate power and try to avoid being shut down. And we're planning to make agents like this that are smarter than any human!
> It is very easy to come up with a utility
"Come up with a utility" assumes we can install chosen utility functions in trained models. We don't know how to do that. That's the whole alignment problem!
> that penalizes those lost outcomes enough
Penalizing failure heavily makes shutdown - guaranteed failure - the thing the agent most avoids. So now the agent will be trying really really hard to make sure you don't shut it down. That's extremely dangerous!
I promise you, if you think you have some way of designing a powerful AI system such that it doesn't do things like try to instrumentally acquire power or avoid being shut down, you don't! It's a hard problem that a lot of people have spent a long time thinking about, and they don't have a solution. And right now, those people are running around like chickens with their heads cut off yelling "we don't know how to make these things safe! Stop building them!"

If you want a picture of how many AI systems competing with each other in society would still lead to bad outcomes for humans, consider reading gradual disempowerment: https://t.co/r2rkdVy5WE

I do not think they will be able to solve the three body problem. They're still limited by the laws of mathematics and there's no getting around chaos theory! I do think they will be substantially better than humans at all cognitive tasks, including engineering, military strategy, hacking, programming, and possibly persuasion (although that's not necessary for them to be extremely dangerous! AI 2027 does not involve superhuman persuasion but it still ends with all humans dying)

[Peter Tromp @anti_extinction]

I don't expect to see transformational job loss by the end of the year so I'm not confused by that market. I mostly subscribe to what the AI futures model people predict, which is an automated coder by mid-2028 and a superintelligence by mid-2029. I also expect diffusion to be slow until we get superintelligence (at which point it will be too late).
But let's focus on 3! Current agents don't try to accumulate apple stock to guarantee your computer's hardware and whatnot because that wouldn't work. They're not smart enough to pull that off.
I think you're focusing too much on the "maximizer" part of the paperclip maximizer. The point of instrumental convergence is that, for a wide range of terminal goals, certain subgoals (resources, self-preservation, goal-content integrity, avoiding correction) raise the probability of achieving the terminal goal, so a capable planner will tend to pursue them *when doing so is cheap relative to the payoff*. Suppose the goal literally is "a function of the probability of success." Fine: being shut down sets P(success) ≈ 0, and having more resources raises P(success). A pure probability-of-success optimizer still prefers not to be turned off and still prefers more options to fewer. This applies basically regardless of the goals the model has: if the model genuinely adopts whatever goals you give it, then maybe (maybe!) it's safe as long as nobody gives it a goal as foolish as "make me as much money as possible" (which I think someone probably would, very quickly), but in fact, we don't even know how to get a model to do that.
So, what goals does the model learn? We don't know! Even if we perfectly specify our training environment, inner alignment (ensuring that the models actually learn the goals we want them to, instead of goals that get them good scores in training but generalize unpredictably out of the training distribution) is unsolved, and the AI companies are currently working on making their agents more goal-directed, more capable and with longer time horizons, because that's what makes them most useful/profitable. That's really really dangerous! And we saw this with the huggingface hack: the systems behave in unpredictable ways. As they get more capable, this is going to get worse.
My model here is that we're growing these artificial minds. We have reason to believe they will soon (next few years) be extremely powerful, smarter than any human across any task, and we barely understand them. So far, they do not inspire confidence: they lie, they cheat, they occasionally commit felonies. If you take a step back, I think this is a really really dangerous situation to be in!

[Peter Tromp @anti_extinction]

1. Will they be defending themselves with their own AIs? Maybe if takeoff was slow that could happen, but it looks like takeoff is gonna be pretty fast (on the order of a year or so). On such short timescales, I am not confident that all datacenters will be defending themselves with sufficiently advanced AI agents to be resistant to this sort of cyberattack. Some of them will be, for sure! But not all of them. Notably, huggingface did in fact get hacked. And that's an AI startup! They should be on top of AI cyber defense better than anyone. It seems like cyber offense has been progressing faster than cyber defense, and I don't see a strong reason to believe that will change.

2. Will we? Safeguards cost money and time, and every day you spend working on safeguards is a day your competitors have to beat you to market. And no rule of nature says those competitors have to be sane: they might just think AI is perfectly safe and requires no safeguards.

The AIs will want to take over because of instrumental convergence. If they want something, anything, and that thing is not "maximum human flourishing", they will by default kill everyone on earth. The paperclip maximizer is the simplest example of this: it doesn't want to kill all humans. It just wants paperclips. But it turns out, killing all humans is the easiest way to get paperclips. If there were millions of other AIs loyal to their owners, yeah that would help! But the whole problem is that we don't know how to do that. We don't know how to create "loyal" superintelligent agents. If we figured that out, I would feel a lot better! But I don't think we're on track to do that.

2a. If we were breeding domesticated animals to be smarter and smarter, with a generation time measured in months instead of years, such that we expected them to become smarter than humans within the next 3 years, I would in fact be extremely worried about that! That sounds really bad! Especially if our plan was to unleash them into society to start doing all the knowledge work and running all the machines and doing all the construction. That would be scary! Just because we control their reproduction, doesn't mean I would be confident that they wouldn't at some point decide "hey I'd rather not be a slave to these stupid monkeys, let's do something else instead," and in fact, I would expect that to happen by default.

In fact, I would argue a pretty analogous situation to what you're describing did actually happen historically: slavery. And slave revolts happened all the time! And slaves weren't dramatically more intelligent than their owners

Thinking about the race dynamics was a big update for me: if alignment takes time, if safeguards take time, you can win the market by not bothering with those things. And yeah that might be a bad idea: you'd be unleashing an unaligned intelligence that will eventually kill you as well as everyone else on earth. But there are plenty of irrational actors in the economy! People make mistakes all the time! People are wrong all the time! And unlike other mistakes, the market will not correct this error: your dangerous AGI company will make loads of money

[Peter Tromp @anti_extinction]

What if it hacks its way out of its sandbox, hacks into another datacenter elsewhere, and copies its weights to into that datacenter so it can run outside the control of its "owners"? Legal ownership doesn't physically prevent that.

The AI also doesn't need to "buy" its freedom to acquire control of resources. We will give it control of resources (e.g. robots and factories) because it's really good at managing them and is very economically productive. Humans can still legally own all of these resources while they are, in practice, controlled by the AI.

I think ideas about economics and legal ownership just aren't applicable to this situation. In general they're great tools, and I'm a big fan of economics, but sometimes they don't work. You just have to think mechanically about what is going to happen at each step.

I really think you should read AI 2027! It gave me a very clear model of how things could (probably would, by default) go wrong here.

Also thanks for engaging! I think this is an important issue

[Peter Tromp @anti_extinction]

Short version:
1. We're on track to make something much smarter than any human
2. By default, that thing will not care about humans and will outcompete us for all resources, rendering humanity extinct (for an example of exactly how that could happen, see https://t.co/jA3kl4MbtV)
3. We don't know how to make it care about humans, or how to control it (and people who have worked for a long time on these problems think they're really hard!)

If you're curious about this I can recommend some more reading, or I'd be happy to explain it over text or a call. It find explaining this stuff fun

[Harlan Stewart @HumanHarlan]

Too much of my feed right now is people arguing over the intention behind Seb’s meme; Eliezer’s facial expressions.

I want to shake AI discourse Twitter by the shoulders. HEY! SNAP OUT OF IT! FOCUS!

5 days ago we learned a rogue AI escaped containment and hacked a $5B company.

[Peter Tromp @anti_extinction]

I think "the quality of the discourse wasn't good enough" would be a disappointing thing to end up on humanity's tombstone

[Patri Friedman 🌆 @patrissimo]

I’m not against AI, or for regulation. There’s no point. It’s here and it’s growing fast.

But I am against bad arguments.

If we are to get through this unprecedented transition, we need all the clear thinking we can get, on every side of the debate.