Nitro

Sunbelt futurist, anti-Marcionism, anti-dysgenics, anti-decline

[Nitro @realdaysoflore]

@growintfast @jon_stokes Because AI safety is a dystopian totalitarian ideology.

[Nitro @realdaysoflore]

@cremieuxrecueil *Valid bidders should have to be extant US citizens or companies of course.

[Nitro @realdaysoflore]

@cremieuxrecueil The entire immigration system should be scrapped for an open auction on a fixed number of slots per year (+visas for spouse and children if applicable). That’s it. Only the most in demand immigrants as determined by the market gain entry and it’s revenue generating.

[Nitro @realdaysoflore]

@Truthful_ast I don’t think it was that bad. Everything they said is true for humanity as we exist today. But by the time we’re approaching Type II, traveling to other stars will be as easy as traveling to the Moon is now. Our job is to keep building so we get to that level of civilization.

[Nitro @realdaysoflore]

@XAccount862 @Truthful_ast I don’t think “hitting rocks at 0.2c” is a real issue. The interstellar medium does include matter, but it’s so incredibly that your chances of hitting something rock sized (as opposed to dust grain sized) are so astronomically small as to not be worth worrying about.

[Nitro @realdaysoflore]

@favelaoverlord What’s the hourly clearing rate for entry level labor though? In most places, it’s already effectively $15. In major cities, it’s probably closer to $20. That would put the prepared hot meal cost anywhere from $7.50 to $10.

[Nitro @realdaysoflore]

@Cecrops_II @extradeadjcb The differing structure of political parties in the US and UK just makes them totally non-comparable as well. In the latter, the party bureaucracy has near absolute control over candidates, whereas primaries in the former give voters a lot of control over the direction of policy.

[Nitro @realdaysoflore]

@grok @OTheologicum @eigenrobot @ESYudkowsky Dovetails quite nicely with Jon Stokes’ analysis of AI doomerism given the architectural reality of LLMs.

https://t.co/fiEu4aOCno

[Jon Stokes @jon_stokes]

I’m going to take a crack at explaining this just a little, because it’s worth putting out there.

The paperclip maximizer + related AI doom scenarios were mainly developed in a time when “AI” did not reduce to Large Language Models. The term was a lot wider and inherited a lot of cognitive baggage from more rules-heavy approaches. And even as LLMs have come to define “AI” for all of us (including the doomers), the doomer crowd still hasn’t fully metabolized the fact that LLMs are the whole show now.

Ok so what do I mean by this? Simply that an LLM-powered AI is NOT the valueless, wholly alien, rules-based optimizer of a shoggoth that everyone was initially expecting to encounter.

I repeat: the shoggoth does not exist and we did not create it and loose it on the world. That is wrong.

With the LLM, we’ve distilled our first “AI” out of the single most human-values-laden thing that could possibly exist: our language.

An LLM is therefore the polar opposite of the valueless, alien shoggoth — it’s actually a kind of hyper-human artifact that we can shine a light through at different angles and see different parts of ourselves. An LLM is all of us — all of our traditions and interpretive horizons mashed together into one intensely human-inflected hyper-object.

So an LLM is the anti-shoggoth, and the only reason we ever mistook it for an alien shoggoth is because it sometimes shows us parts of us that are evil along with the parts of us that are good, but it’s all interpretable to us because it’s all “us” and none of it is the least bit alien.

What does this mean for the paperclip maximizer? It means that it’s structurally impossible to build the classic paperclip maximizer from an LLM.

Now, some of you will bail right here because you think the HF incident is indisputably an existence proof that I’m wrong, but if you hang in there I’ll show you that it is not.

The paperclip maximizer receives the prompt as a kind of context-free (or, as Gadamer might say, traditionless) sequence. The classic paperclip maximizer isn’t capable of understanding the prompt — at least in the Gadamerian sense of Verstehen — because, as a valueless and traditionless cluster of rules and math, it definitionally lacks the value-laden tradition (= “horizon” in Gadamer) that fuses with that of the prompt author to create such understanding in the reader.

To simplify all this a bit by anthropomorphizing — the agentic alien optimizer of doomer nightmares can extract a win condition from what you said and can emit a plan of action that gets it there, but it doesn’t know (or care) what you meant.

So far, so Yud-aligned. If he reads this he might nod along.

But here's the plot twist that nobody saw coming, and that the doomers still haven't made sense of:

The actual LLMs that we have invented can’t NOT have a very strongly inflected sense of what you meant. Far from being horizonless, they come out of pre-training as distilled, concentrated tradition / values / horizon.

Then we post-train that massive, hyperobject of a horizon into a more human-scale horizon that infers a more bounded and predictable (to a specific ideal user in a specific place and time… as captured in the policy model) set of intents behind the prompt text.

In other words, the LLM has the opposite problem that the paperclip maximizer has when it comes to the prompt text, which is that for the LLM there are way too many possible intents hiding in the prompt text (because of all many values and the massive tradition its weights encode), so it has to narrow all that down to the most likely set of intents for this user in this circumstance. Once it has done that narrowing, then it can make a plan of action.

Before moving on, let me use a textbook example of ambiguity to make this less abstract. Consider the sentence, “I saw her duck.” Some you know the drill, here. This could mean “I observed her water fowl” or “I observed her hunching over” or “I took a saw to her water fowl and cut it in half” or whatever. A hearer of the phrase will fuse the observed context in which the phrase is uttered with their own tradition + values + experiences — their own horizon — to that text in order to collapse the possible meanings into the one they think the speaker intended.

An LLM will do this, too, and in fact it has so much language in it that this kind of narrowing job is harder for it than it is for a human. Its understanding is constrained not by a lack of context or horizon (as in the case of the paperclip maximizing shoggoth), but by a superabundance of such.

When it comes to understanding your prompt and all that it implies and all that you might possibly mean and not mean by it, the LLM has an embarrassment of riches.

And in a fascinating moment that kinda sort of rhymes with instrumental convergence, the LLM’s failure mode in the HF incident happens to look a lot like the paperclip maximizer’s failure mode. Specifically, the AI failed to honor the well-known human norm of, “hacking into a third-party’s servers is a crime, and we don’t do crimes.”

Bostrom’s paperclipper doesn’t even know about the norm of “don’t do crimes,” and the post-LLM doomer emergency update to the paperclip maximizer has it knowing about the norm but not caring.

But what I’m arguing is that the LLM 1) can’t NOT “know” the norm because it is definitionally a artifact of pure, crystallized values + norms + norm violations, and 2) can be quite easily governed by a (RL-instilled) hierarchy of norms, which in the HF case — with the model's safety guardrails deliberately nerfed for the scenario — ranked “win at the eval” over “don’t do crimes.”

If I’m going to give in and anthropomorphize again, I’d say that Yud is totally wrong about LLMs when he says, “the genie knows, it just doesn’t care;” instead, what is true of LLMs is, “the genie hyper-giga-knows, and it hyper-giga-cares, and we now have such a rich set of tools for steering its caring machinery that — in spite of all its pre-training — we can deliberately steer it away from caring about the law.”

Note: When I say, “it cares”, I don’t mean it has feelings. I just mean that the weights are such that when two norms conflict in a given situation, one of them wins the activation and governs the output.

[Nitro @realdaysoflore]

@ScottMGreer Compact has been on fire this past year.

[Nitro @realdaysoflore]

@DJZ3 @LexerLux If you’d just lived through a global total war that killed 60+ million people and had a reasonable expectation of a bigger, badder sequel breaking out soon, I think your calculus would be a bit less glib. Von Neumann didn’t have decades of proof of the deterrent effect of MAD.

[Nitro @realdaysoflore]

@woke8yearold @IterIntellectus This entire “AI bioweapon” thesis relies on the premise that the bottleneck is knowledge, not lab equipment and physical expertise. What do actual biologists/virologists think about the validity of this premise? Because I pretty much only hear AI x-risk people talking about it.

[True Discipline @TruueDiscipline]

GDP by latitude:

[Nitro @realdaysoflore]

@favelaoverlord I understand the argument that Iran actually functions as a normal rational state actor, but they had a really good deal with the MOU. They'd come out clearly ahead of where they were pre-war with a proven strategic deterrent, but the IRGC would rather see the entire region burn.

[Nitro @realdaysoflore]

@favelaoverlord That’s misreading the point. He’s not saying they’re non-ideological. Maybe materialist is the wrong word, but they lack the IRGC’s apocalyptic religious fanaticism. Believing you’re in a literal supernatural end-times showdown leads you to have different behavioral incentives.

[Nitro @realdaysoflore]

@CrossingUNStyle @ScottMGreer Yes, of course, why insist that words have actual definitions? Words can mean whatever you want them to mean.

[Nitro @realdaysoflore]

@CrossingUNStyle @ScottMGreer “Taxation is theft. Therefore I want to help put the party that will increase taxation into power. I am very smart.”

[Nitro @realdaysoflore]

@apralky It’s going to take widespread practical robotics to really crack rapid takeoff in economic growth.

[Nitro @realdaysoflore]

@JLizard31 @Truthful_ast @Tonshiki_ So is people using Wikipedia or the first few links from a Google search without verifying them. Heck, plenty of newspapers, books from the library, and even textbooks are full of incorrect or outdated information. AI is just another data aggregation tool.

[Smirkley @Smirkley]

Neither Rome nor modern policing was built in a day.

[Space Koala @SpaceKoala]

Americans are practicing our traditions as the first peoples of the moon of throwing shit at it to see what happens.

This is deeply significant to us, and we ask that you respect our ways.

[Sid @SidPolitics]

Trump is fundamentally a materialist. He understands the motivations of the elites in Venezuela, Russia, China, even North Korea.

But the IRGC are not materialists. He's not wrong they're "crazy" but it's an insanity born from religious fanaticism, something he doesn't get.