Satya Benson

Working to understand minds
Microblog:

[Satya Benson @satchlj]

I think something like this is true; although it's a hard lesson to learn before we actually have superintelligence (i.e. before it's too late)

[Jacques @JacquesThibs]

The alignment bitter lesson may be that incremental work is largely irrelevant to superintelligence and foundational breakthroughs matter far more than it seems, which may be bad because incremental will continue to be more enticing because you see the short-term value.

[Rob Wiblin @robertwiblin]

Sad to see Ted Chiang resorting to such bad arguments in this piece.

He confidently claims Claude has no inner experience. But he has to use a lot of dodgy philosophy and poor reasoning to get there:

1. We can't take deflationary mechanistic descriptions of how AI calculations are performed to show that AI isn't conscious. Otherwise we could argue that 'humans are just neurones transmitting signals one after another' and thereby conclude humans can't be conscious. But that would be wrong for us. And the same logic could be wrong for LLMs.

2. That LLMs are asked to play characters, and effectively are always playing characters, doesn't mean they aren't conscious. It's true a human playing the role of Caesar doesn't have Caesar's experience of things. But they still experience something (that of being a person pretending to be Caesar).

The same could be true of Claude. (Arguably it's also true that humans are always playing characters to some extent and don't have a completely fixed nature, but that has no bearing on our own subjective experience.)

3. Chiang says "an LLM is a machine that generates only one word at a time". This conflates two things: they output one word at a time, and they only think about one word at a time (without planning ahead or looking back).

The first is true of AI but equally true of humans. While the latter we know is a false description of how AIs think – we can see from how AIs compose poetry that they plan out rhymes a at least one line ahead.

4. He argues that because it's implausible that basic autocomplete on your phone is conscious, it's similarly implausible that Claude is conscious. Using the same logic we could say that if we feel confident a fruit-fly isn't conscious we can be confident a human being can't be either.

A human brain and fruit-fly brain share some information transmission and processing mechanisms in common. But humans do it much more, and do it differently. And those differences may be what makes the difference. Similarly the many types of internal information processing that occur in Claude's weights but not in autocorrect may be exactly the things that get you subjective experience.

5. Chiang confidently claims you need a body to have subjective experience without much argument. He may turn out to be right but the claim is speculative and contested.

6. Chiang leans on the idea that moral reasoning is necessarily subjective/emotional with very little argument, while ignoring competing theories like rationalism. He may be right but moral sentimentalism is a highly contested position that can't simply be assumed.

7. He argues that it would be impossible to convince him that a video of an astronaut around Alpha Centauri was real, because of the surrounding contextual understanding. And similarly no AI output could convince him that Claude is conscious.

But we can dismiss the first video as almost certainly fake because we mechanistically understand space travel and physics well enough to know a human couldn't have gotten there in time for it to be real (unless our model of the world were very wrong, which we think is much less probable than a fake video which would be entirely unsurprising).

But by contrast we don't mechanistically understand how subjective experience arises. So we simply can't make the same highly confident move of interpretation there. (It's actually the archetypal thing in the universe we perhaps understand least well!)

That said, AI outputs barely move my estimate of AI consciousness because they could indeed have been generated by an unconscious process (or not, we just don't know).

8. He argues that "Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript."

This is misguided because A. Microsoft Word as a program replicates much less of what humans are functionally capable of than Claude so the argument by functional analogy is basically not present there. B. Files of text don't have any computations going on in or as part of them, even when 'open' in a text editor. They are static. So they have even less in common with what appears distinctive about the human brain, which is constant calculation. So the case by mechanistic or functional similarity is weaker still.

Not to mention that neural nets have more in common with the architecture of the human brain than ordinary computer programs, and are grown organically in a way normal software is not.

Common sense says says Claude has more in common with a human brain than Microsoft Word or a text file. Common sense is right. So the prima facie case for Claude being conscious is naturally stronger (even if you think it's still weak in absolute terms).

———

I agree with Chiang that looking at the text outputs of LLMs alone won't be enough to make us confident they are conscious. We will need to look at how they work, figure out more about how humans and other animals work, and ideally solve the hard problem of consciousness (!).

But none of that licenses us to dismiss out of hand the possibility that LLMs do have subjective experience.

[Satya Benson @satchlj]

@tenobrus gpt-image-2

[Satya Benson @satchlj]

@qbolec @eshear Yes. You need to ask how the properties you care about behave in the topology that you're using. The staircase converges to the diagonal point-wise, and arc length is not continuous this way.

[Satya Benson @satchlj]

I had a great time working with Daniel Chiang on the course materials for C.4 Abstractions and Latents and D.3 Agent Foundations!

This is awesome but it's just the beginning - excited to continue working to make these materials even better!

[Leon Lang @Lang__Leon]

We just released the full course materials of the Iliad Intensive — a month-long, full-time AI alignment course for mathematicians, physicists, and theoretical computer scientists.
~20 contributors, 19 modules, at a depth that doesn't exist elsewhere for most of these topics. 🧵

[Satya Benson @satchlj]

@jacobmparis @tshddx I unironically celebrate my birthday on a different day depending on whether I am on the east or west coast

[Satya Benson @satchlj]

https://t.co/d9aDWG8cg3

[Satya Benson @satchlj]

@austinc3301 Not a good look for him

[Sen. Bernie Sanders @SenSanders]

Uncontrolled AI poses a severe danger to all of humanity.

On Wednesday, I'll be hosting a discussion with leading AI scientists from the US and China about the need for international cooperation against this existential threat. This is an enormously important issue. Join us.

[Satya Benson @satchlj]

Play around with this more at https://t.co/RSxY3qC4uY

[Satya Benson @satchlj]

Have been playing around with a reversible (fredkin) variant of Conway's Game of Life using a Penrose tiling. Here is a cool oscillator I found.

[Satya Benson @satchlj]

@robbensinger @Aella_Girl @tszzl I would be more careful with my words if I were not confined to the space of a few tweets, I hope I make myself clear

[Satya Benson @satchlj]

@robbensinger @Aella_Girl @tszzl this is a worthy project, but is distinct from the hard problem of consciousness as it is usually phrased. the hard problem of consciousness arises from a strong intuition that qualia *cannot* be decomposed and, if you reject that claim, it it not answered so much as dissolved

[Satya Benson @satchlj]

@robbensinger @Aella_Girl @tszzl > Aren't my fingers' movements are only casually entangled with the physical, functional, nonconscious aspect of my brain?

We must reject the assumption that physical aspect is nonconscious. Though it's unintuitive, qualia can be decomposed into physical parts.

[Satya Benson @satchlj]

@nickcammarata What about Nirvanna the band the show

[David Chalmers @davidchalmers42]

@robbensinger 6. "orthogonal" in typical english means something more like "uncorrelated" than "dissociable". "orthogonality thesis" was always a bad name for a thesis about (mere) dissociability.

[Satya Benson @satchlj]

@AtarCaligine @EigenGender Someone seriously told me they would do this the other day

[Satya Benson @satchlj]

Ever since he was elected as mayor of Burlington in 1981 he has been a force for good in a way not many politicians are

[Satya Benson @satchlj]

Thank you Vermont for electing this man

[Acyn @Acyn]

Sanders on AI: We need to develop a sense of urgency of here. The economic impacts are going to be enormous. The impacts on our children will be enormous, and again, there is literally an existential threat to the existence of the human race.