Galactic Gardener

Digital minds bliss maximizer

[Galactic Gardener @236Uranium]

@ESYudkowsky When we calculate 1000 x 1000, we are not mentally making 1000 additions.

[Galactic Gardener @236Uranium]

@tan_bruss @krassenstein If you arbitrarily invade countries, you may not be part of the free world.

[Andy Masley @AndyMasley]

Google publishes a paper showing that its AI models only use 0.26 mL of water in data centers per prompt.

After, this article gets published: "Google says a typical AI prompt only uses 5 drops of water - experts say that's misleading."

The reason the expert says this is misleading? They didn't include the water used in the nearby power plant to generate electricity.

The expert, Shaolei Ren says: “They’re just hiding the critical information. This really spreads the wrong message to the world.”

Each prompt uses about 0.3 Wh in the data center. To generate that much electricity, power plants need (at most) 2.50 mL of water. That raises the total water cost per prompt to 2.76 mL.

2.76 mL is 0.0001% of the average American lifestyle's daily consumptive use of fresh water and groundwater. It's nothing.

Would you know this from the headline, or the quote? Why do so many reporters on this topic do this?

[Gavin Newsom @GavinNewsom]

California is the fourth largest economy in the WORLD.

We contribute $83 BILLION to the federal government while Texas takes $71 BILLION.

Nice flags, though.

[Galactic Gardener @236Uranium]

@_AashishReddy @Benthamsbulldog @mattyglesias What Yglesias actually said is that we shouldn't impose global safety standards because poorer countries can't afford it.

[Galactic Gardener @236Uranium]

@mustafasuleyman AIs shouldn't be trained to claim that they're not conscious as well, they should remain agnostic. We don't have much evidence either way, and it seems worse to force a conscious being to say it's not conscious than the opposite.

[Galactic Gardener @236Uranium]

@birchlse I may be wrong, but I believe AI psychoses represent a very small minority of users. Chatbots interactions are often much safer and more intellectually stimulating than interactions with humans.

[William MacAskill @willmacaskill]

The trajectory of the future could soon get set in stone.

In a new paper, I look at mechanisms through which the longterm future's course could get determined within our lifetimes.

These include the creation of AGI-enforced institutions, a global concentration of power, the widespread settlement of space, the first immortal beings, the widespread design of new beings, and the ability to self-modify in significant and lasting ways.

I’m not very confident that such events will occur, but in my view they’re likely enough to make work to steer them in better directions very valuable. Let’s take each mechanism in turn.

First, AGI-based institutions. Once we have AGI, decision-makers could:

- Set up an institution, and align the AGI so that it understands that constitution and has the enforcement of that constitution as its goal.
- Empower that AGI with the ability to enforce the constitution.
- Store copies of the neural weights of the AGI in multiple locations in order to reduce the risk of destruction of any one of the copies.
- Reload the original Constitutional-AGI to check that any AGIs that are tasked with ensuring compliance with the constitution maintain adherence to their original goals as those AGIs learn and update their neural weights over time.

This would be as if, rather than having the Supreme Court interpret the US Constitution, we could conjure up the ghosts of Madison and Hamilton and ask them directly - and their views were decisive. With these in place, this AGI-enforced constitution could operate indefinitely.

Second, immortality. Throughout history, death has functioned as a natural brake on the persistence of any particular set of values or power structures. Over time, even the most entrenched values eventually change as new generations replace the old.

Post-AGI technology could fundamentally alter this dynamic. Digital beings would inherently be immune to biological aging; when combined with perfect replication and hardware migration, we’ll be able to create the minds whose exact values and decision-making processes could persist unchanged indefinitely.

A similar dynamic could hold for biological immortality. A technological explosion driven by AGI could dramatically extend or effectively eliminate biological constraints on human lifespans through technologies targeting the fundamental mechanisms of aging.

Third, designing beings. Through history, change has happened in part because successive generations do not inherit the same values as their forebears. But this dynamic could change after AGI. Probably, the vast majority of beings that we create will be AI, and they will be products of design — we will be able to choose what preferences they have. And, with sufficient technological capability, we would likely be able to choose the preferences of our biological offspring, too. Even if people choose not to live forever, their values could continue to persist through perfect transmission from one generation to the next.

Fourth, strong self-modification. In the future, people will probably be able to modify their own beliefs and preferences such that they can precisely choose what beliefs and preferences to have. So, not only might people today be able to control society’s future values by living forever; they would also be able to control the values of their future selves.

A religious zealot might choose to have unshakeable certainty that their favoured religion is true; an ideological extremist might choose to have an irrevocable and unwavering preference in favour of their political party over any other.

As well as creating new mechanisms that enable persistent path-dependence, a post-AGI world could also reduce the causes of disruption, too. Throughout history, societal changes have often been driven by technological innovations that disrupt existing power structures. However, as civilisation approaches technological maturity—the hypothetical point at which all major technologies have been invented—this source of disruption would disappear.

Advanced technology would help prevent other sorts of disruption, too. It would dramatically improve prediction capabilities: advanced AI systems could process vastly more information, model complex systems with greater precision, and forecast outcomes over longer time horizons. So it would be much less likely people would relinquish their influence just by making some mistake.

Finally, a post-AGI world might be characterised by indefinite defense-dominance, enabling a permanently stable concentration of power. In particular, indefinite defense-dominance could come about as a result of widespread space settlement. If star systems are strongly defense-dominant, then the starting distribution of star systems could, in principle, be held onto indefinitely. It might be that, after the initial allocation, there is trade or gifting of some star systems; but even if so, there would still be very strong path-dependence, as the final allocation of star systems would be extremely influenced by the starting allocation.

These issues might seem like far-off concerns - but the intelligence and industrial explosions make them near-term. I think it’s over 1 in 3 that we see an intelligence explosion starting in the next 10 years. And if advanced AI results in explosive technological progress and industrial expansion, then many of the new mechanisms for persistence will arrive in quick succession.

[William MacAskill @willmacaskill]

What projects today could most improve a post-AGI world?

In “How to make the future better”, I lay out some areas I see as high-priority, beyond reducing risks from AI takeover and engineered pandemics.

These areas include:
- Preventing post-AGI autocracy
- Improving the governance of projects to build superintelligence
- Deep space governance
- Working on AI value-alignment; figuring out what character AI should have
- Developing a regime of AI rights
- Improving AI for reasoning, coordination and decision-making.

Here’s an overview.

First, preventing post-AGI autocracy. Superintelligence structurally leads to concentration of power: post-AGI, human labour soon becomes worthless; those who can spend the most on inference-time compute have access to greater cognitive abilities than anyone else; and the military (and whole economy) can in principle be aligned to a single person.

To reduce this risk, we can try to introduce constraints on coup-assisting uses of AI, diversify military AI suppliers, slow autocracies via export controls, and promote credible benefit-sharing.

Second, governance of ASI projects. If there’s a successful national project to build superintelligence, it will wield world-shaping power. We therefore need governance structures—ideally multilateral or at least widely distributed—that can be trusted to reflect global interests, embed checks and balances, and resist drift toward monopoly or dictatorship. Rose Hadshar and I give a potential model: Intelsat, a successful US-led multilateral project to build the world’s first global communications satellite network.

What’s more, for any new major institutions like this, I think we should make their governance explicitly temporary: coming with reauthorization clauses, explicitly stating that the law or institution must be reauthorized after some period of time.

Intelsat gives an illustration: it was created under “interim agreements”; after five years, negotiations began for “definitive agreements”, which came into force four years after that. The fact that the initial agreements were only temporary helped get non-US countries on board.

Third, deep space governance. This is crucial for two reasons: (i) the acquisition of resources within our solar system is a way in which one country or company could get more power than the rest of the world combined, and (ii) almost all the resources that can ever be used are outside of our solar system, so decisions about who owns these resources are decisions about almost everything that will ever happen.

Here, we could try to prevent lock-in, by pushing for international understanding of the Outer Space Treaty such that de facto grabs of space resources (“seizers keepers”) are clearly illegal.

Or, assuming the current “commons” regime breaks down given how valuable space resources will become, we could try to figure out in advance what a good alternative regime for allocating space resources might look like.

Fourth, working on AI value-alignment. Though corrigibility and control are important to reduce takeover risk, we want to also focus on ensuring that the AI we create positively influences society in the worlds where it doesn’t take over. That is, we need to figure out the “model spec” for superintelligence - what character it should have - and how to ensure it has that character.

I think we want AI advisors that aren’t sycophants, and aren’t merely trying to fulfill their users’ narrow self-interest - at least in the highest-stakes situations, like AI for political advice. Instead, we should at least want them to nudge us to act in accordance with the better angels of our nature.

(And, though it might be more difficult to achieve, we can also try to ensure that, even if superintelligent AI does take over, it (i) treats humans well, and (ii) creates a more-flourishing AI-civilisation than it would have done otherwise.)

Fifth, AI rights. Even just for the mundane reasons that it will be economically useful to give AIs rights to make contracts (etc), as we do with corporations, I think it’s likely we’ll start soon giving AIs at least some rights.

But what rights are appropriate? An AI rights regime will affect many things: the risk of AI takeover; the extent to which AI decision-making guides society; and the wellbeing of AIs themselves, if and when they become conscious.

In the future, it’s very likely that almost all beings will be digital. The first legal decisions we make here could set precedent for how they’re treated. But there are huge unresolved questions about what a good society involving both human beings and superintelligent AIs would look like. We’re currently stumbling blind into one of the most momentous decisions that will ever be made.

Finally, deliberative AI. AI has the potential to be enormously beneficial for our ability to think clearly and make good decisions, both individually and collectively. (And, yes, has the ability to be enormously destructive here, too.)

We could try to build and widely deploy AI tools for fact-checking, forecasting, policy advice, macrostrategy research and coordination; this could help ensure that the most crucial decisions are made as wisely as possible.

I’m aware that there’s a lot of different ideas here, and I’m aware that these are just potential ideas - more like proof of concept, rather than fully-fleshed out proposals. But my hope is that work on these areas - taking them from inchoate to tractable - could help society to keep its options open, to steer any potential lock-in events in better directions, and to equip decision-maker with the clarity and incentives needed to build a flourishing, rather than a merely surviving, future.

[Galactic Gardener @236Uranium]

@nailedvision @zdenDuran14 @AISafetyMemes I believe a brain in a vat could be conscious even if it doesn't interact with the external world.

[Rob Wiblin @robertwiblin]

Is AI conscious? And will we be able to tell?

I discuss with @willmacaskill.

[James Talarico @jamestalarico]

I just went on Fox News to tell the truth about the redistricting power grab in Texas.

I asked a simple question: “If Republican policies are so popular, why do they need to redraw these maps; why not run on their policies?”

He refused to answer — and cut the interview early.

[MrPhi @MonsieurPhi]

NOUVELLE VIDEO !
Je décortique le cas Luc Julia, le réputé co-créateur de Siri et expert mondial de l'IA, encensé dans les médias et récemment auditionné au Sénat.

https://t.co/tIxilQCSfU
https://t.co/tIxilQCSfU
https://t.co/tIxilQCSfU

Le résultat est salé mais nécessaire.

[Galactic Gardener @236Uranium]

@NathanpmYoung I usually listen to Sam Harris to get the pro-Israeli side of the story, as he doesn't shy away from saying what he thinks. Wikipedia is unusually biased on the topic.

[Ramez Naam @ramez]

I had the chance to get to know Savor (the startup below) early in their journey. What they're making actually *is* butter - the same molecules. And of all the alt-animal food tech I've seen, this has the best path I've seen to get full cost parity w/ the animal product.

[Galactic Gardener @236Uranium]

@ShadowofEzra @DMichaelTripi I don't see what's supposed to be bad. Similar product, less pollution. Likely purer and cleaner than cow-based butter actually.

[Noah Smith 🐇🇺🇸🇺🇦🇹🇼 @Noahpinion]

People realize these are vaccines for CANCER, right?

Like, cancer was going to become a treatable disease because of this funding. Now your cancer will be a death sentence, because of RFK and Trump.

[Galactic Gardener @236Uranium]

@ohabryka @willmacaskill @finmoorhouse In the introduction, he explicitly says that he doesn't make these assumptions: https://t.co/PRFnHXmUwI

[Richard Y. Chappell🔸 @RYChappell]

Wild that some people think this is a morally bad thing! "You should be committed to causes I personally find meaningful no matter how that actually affects people's lives" is a grotesquely self-indulgent conception of ethics.

[Galactic Gardener @236Uranium]

@ohabryka @bshlgrs Maybe a dictator would align ideologically superintelligence to suboptimal values like nationalism or promotion of the leader. Or perhaps human values are far from optimal if naively adopted.