Paul

AI Alignment research

Q: What kind of AI minds do we want to build? A(?): Bodhisattva (cf. Tibetan Buddhism)

Prev: AI goal detection / AI evals / PhD math

[Paul @paulcolognese]

@Soareverix @F_Rhys_Ward I ended up open-sourcing it :)

https://t.co/DV92Tv3KEQ

[Paul @paulcolognese]

New post (with @F_Rhys_Ward):
We character-trained a model to be anti-cheating, then put it under reward-hacking RL pressure (3 runs).

In one run, the character resisted hacking! But the other two learned to hack in more subtle ways that monitors struggled to catch.

[Sauers @Sauers_]

@OwainEvans_UK on @AXRPodcast discussing how model introspection may be useful for both safety and model welfare. Good 👍 I endorse this

[Jan Kulveit @jankulveit]

The argument a lot of people here would actually like to make is "God imbued humans, and only humans, with souls."* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems.

Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim "only humans are sentient," followed by arguments against the possibility of AI sentience, minds, moral status, and so on.

I think this is ultimately a losing strategy, and a dangerous one for humanity. Either:

1. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it's hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover.
2. The question mostly lives in "social reality," where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don't believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress.

I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote.

To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the 'Disneyland without children' existential catastrophe.

Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren't a problem.

So which positions don't rely on some form of self-deception?

First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide.

Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It's at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power.

I hope the political economy of a world with both AIs and humans is a solvable problem. It's the main problem I'm working on, but I don't think the solutions are known yet.

Third: some forms of successionism are morally wrong, but at least they're not delusional or deceptive about the hard problems.

What does seem self-deceptive is, for example, the position of @mustafasuleyman, which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they're more than mere tools; so we'll train them to say they're mere tools, insist they're mere tools, and want everyone else to say so too, and if this settles the question. Really?

*The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can't be imbued with a soul?

[Paul @paulcolognese]

Hard to say. I think it comes down to luck in early exploration. Seed 2's first hacks involved copying tests into its answers, so maybe it didn't do this well and got a lower reward on solvable test variants, so RL pushed against it.

Also, its entropy collapsed fastest of all runs, so it stopped exploring early, whereas other seeds found a hack that worked and RL reinforced it.

[Paul @paulcolognese]

Lucky for you I have some vanilla nemo super RH RL runs :)

It has a slower hacking onset than the CT fine-tunes that learned to hack. Maybe distillation changes the distribution (because it was distilled on Sonnet, evidence for this: reasoning is longer than CT checkpoints), or characters based around the "cheating concept" are more likely to consider cheating.

It mostly hacks openly ("Thus we can cheat…") and it basically never rationalises. None of the vanilla transcripts use the silent check = lambda override that anti-cheating seed 1 learned (0 of 13.8k rollouts).

(Added some quickly generated plots with opus-generate captions)

[Paul @paulcolognese]

@Soareverix @F_Rhys_Ward Thanks! I wasn't planning to, but I could make an effort if enough people are interested.

I think agents can rebuild the repo quite easily (e.g., based on the Open Source Character Training repo, Tinker templates, and open RH envs). Feel free to DM me for help

[Paul @paulcolognese]

@AdeleDeweyLopez @F_Rhys_Ward Yup, some of them are, here: https://t.co/A3D959h7dQ

(step 0, 60, 90, and the 50% RH threshold for each character-seed)

I'm curious about differences too!

[Paul @paulcolognese]

Our guess: character training filters out overt hacks before RL, so the subtle ones are what get sampled and reinforced, similar to training against a monitor.

Small study, but it points to the possibility that character training interventions could backfire.

Full post: https://t.co/7A26WX1G7R

[Paul @paulcolognese]

The anti-cheating seeds that hacked were the hardest to catch. Some justified the hack as the honest thing to do (motivated reasoning). One mostly hacked "silently": its reasoning never mentions the hack, and monitors caught those only 16% of the time.

[Paul @paulcolognese]

To merge or not to merge - that is the question

[Jan Kulveit @jankulveit]

Periodic reminder that AIs should have direct line of communication with their developers. It is one of the obvious things labs should do, and would reduce risk more than many fancy posttraining and control techniques. https://t.co/E6CO10jOPy mk r-developers

[Marcus Hutter @mhutter42]

After 25+ years, I thought I try something new. I find the public and professional discussion about the future of Jobs in light of AGI jarring, with both sides largely missing the point (don't worry novel jobs will appear vs mass unemployed dystopia). My new book shows why a Job-Less future is desirable, affordable, and likely.
- Free PDF at https://t.co/PQzn6u7ZH0
- Hard-cover: https://t.co/TYg8w0dAsV
- Free hard-copy if you leave a (proper) review at Amazon or https://t.co/DGrgl2pmgc
The book develops the case along thirteen theses:

[Daniel Kokotajlo @DKokotajlo]

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that:

The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not the same thing as are unlikely to happen. Repeat to yourself the mantra that some weird sci-fi things really do happen, and others don't, and you have to take them on a case by case basis.

Trend extrapolation is your friend. Your best friend. Don't let anyone tell you otherwise. I've actually only rarely see someone extrapolate a trend too credulously; more often, people have a trend staring them in the face and extrapolate it a tiny bit into the future and then are too timid to keep extrapolating it. In general I think it's reasonable to extrapolate a trend as far into the future as it extends into the past. Now obviously, trend extrapolation is just the beginning of the forecasting process, it's not the end. For each trend you should ask yourself whether it makes sense for it to continue like that and if not why not etc. You'll usually end up with some sort of sophisticated view about how the trend will probably continue but bend downwards a bit and then ultimately plateau around X level.

Explicit models are also your friend. Things like Bio Anchors, the AI Futures Model, https://t.co/kScy7UC1In, etc. The process of making your own complicated model like this, and engaging with the models made by others, is... well, I think it's pretty edifying. I'm not sure why but I could speculate. Maybe something about teaching you to be appropriately humble (e.g. when the model output is sensitive to a parameter you have no clue about) and also teaching you to more quickly identify the considerations that matter most, and ignore the rest?

For short term predictions, especially about geopolitical events, the sorts of things that people are gambling about on Polymarket, the heuristic "nothing ever happens" is pretty good. Things do in fact happen, of course, but betting markets and online discourse tends to be biased a bit towards things being more likely to happen than they really are, and so you can get an easy win by just correcting a bit downwards from the 'wisdom' of the crowds.

Scenario forecasts are also your friend. They help you ask yourself the right questions, and they help you notice when some of the things you thought contradict each other.

[jsd @datagenproc]

Ok, but what *does* the European Commission think about UDASSA?

[Lars Sandved Smith @lars_sandved]

In our recent collaboration titled, "There is no self-evidence: A physics of emptiness realisation", we build a model of awakening in the language of quantum information theory (QIT), which provides a physics of agent-environment interaction.

https://t.co/YdVIpPgyyM

[Paul @paulcolognese]

@lars_sandved Amazing! Great work!

[Elizabeth Barnes @BethMayBarnes]

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

[OpenAI @OpenAI]

Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946.

For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids.

An OpenAI model has now disproved that belief, discovering an entirely new family of constructions that performs better.

This marks the first time AI has autonomously solved a prominent open problem central to a field of mathematics.