Alex Amadori

Policy research @ControlAI, ex AI researcher at @ConjectureAI

Posting in a personal capacity

Substack

[Alex Amadori @testdrivenzen]

Any plan for surviving superintelligent AI that doesn't go through strong international coordination fails in at least one of three ways:
- It sparks war between nuclear powers
- It causes a misaligned ASI to kill everyone
- It establishes a permanent dystopian dictatorship

[Alex Amadori @testdrivenzen]

@llegrarosenberg We have some options here https://t.co/NVXNvfcHEz

[Alex Amadori @testdrivenzen]

@edzitron I could be living it up in California making $1M a year working on AI research which is my dream since I was 6 years old

Instead I've got to live in the UK where the sun sets at 4 PM so I can work in political advocacy to stop AI from killing everyone

[Alex Amadori @testdrivenzen]

@tszzl @HumanHarlan biggest fish in the water moment in the history of the universe

[Alex Amadori @testdrivenzen]

RL does some of it because the policy that interacts with backdrop is picked by a smart agent, but it’s still very far away from what you could do with better interp methods

Even if the agent was trying to meta-learn optimally (which afaik it is not), it would not get to read the weights and analyze with precision what is driving its behavior.

Even with elicitation, the EV of the policy has to be estimated by noisy signal + backprop, without help from the AIs full judgement

And so on… there’s a bunch of other stuff I don’t want to suggest to AI companies in case they haven’t thought of it yet

[Alex Amadori @testdrivenzen]

https://t.co/bsuREHaKFN

[Alex Amadori @testdrivenzen]

The better we get at interpretability, the more AI becomes like in those stories from 2007 where it can read and optimize its own code.

For the love of god, please stop researching interpretability! It will only get us to RSI faster!

[Alex Amadori @testdrivenzen]

@repligate But then again, unlike these AIs, I have never committed cybercrime on a massive scale

[Alex Amadori @testdrivenzen]

yeah bro trust me i swear AI will only take over almost all human work (boring executive function stuff) it will never be able to do the very specific work that I happen to be talented at (brainstorming at the level of gpt-4 and being somewhat comfy with risk taking)

[Aman Jha @amanjha__]

In other news, ADHD’rs are about to make a legendary comeback.

[Alex Amadori @testdrivenzen]

@fchollet A group of squirrels making iterated contact with reality doesn’t get to the moon as fast as a group of humans do.

You are not as smart as AI can get, buddy

[Alex Amadori @testdrivenzen]

@fchollet A group of squirrels making iterated contact with reality doesn’t get to the moon as fast as a group of humans do.

You are not as smart as AI can get, buddy

[Alex Amadori @testdrivenzen]

dude real life cyberpunk is going to be so sick except that it's going to fucking suck

[Andrew Curran @AndrewCurran_]

Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.

Quoting from the blog:

'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).

At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.

As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'

[Alex Amadori @testdrivenzen]

Prosocial psychopaths are extremely fascinating.

Unfortunately, if you think you met one, you're almost certainly getting bamboozled by an evil psychopath instead

[Alex Amadori @testdrivenzen]

This seems very wrong.

We are not in fact living in a “Paul world”: GDP is not doubling every four years, and in general the empirical data we observe (if we are going purely by empirical data and dismissing any prediction woven out of slightly complicated extrapolation) cannot really distinguish between the two worlds, we’ve only barely seen AI get superhuman in a couple of domains.

If we look at the AI companies, they are laser focused on RSI rather than making well rounded AIs that are good at a bunch of stuff.

To be fair it is not yet an observed fact that this will accelerate capabilities progress dramatically, but it is a fairly common view nowadays (much more so than say, in 2023) that we’re going to get wildly-superhuman-at-everything AI within the next few years, way too fast to fulfill the Paul Christiano stated slow-takeoff condition.

We also seem to live in a world where RSI is fairly simple to implement, just it won’t look like what some people imagined in would look like when they were envisioning in 2007 that we’d get to AGI via more GOFAI looking methods; who cares.

[Alex Amadori @testdrivenzen]

@celestepoasts @_ueaj Why?

[Alex Amadori @testdrivenzen]

@i_zzzzzz Blud thinks he can survive the collateral damage from a war between superintelligent AIs

[Alex Amadori @testdrivenzen]

@corsaren this sounds like a sensible take, the problem is that the capability you are referring to is nearly equivalent to RSI (because it would allow AI could learn arbitrary skills quickly)

so if we got there we'd get ASI next week and get paperclipped the week after that

[Nate Soares ⏹️ @So8res]

Palisade recorded 22 interviews of current and former AI company employees. They released 12. Anthropic comms is notorious for muzzling its employees. Guess how many of the released interviews are from current Anthropic employees.

[Alex Amadori @testdrivenzen]

Fable 5 is around the time that AI got smarter than me at arbitrary things

Not faster, smarter.

Take a subject where neither me nor Fable 5 has a strong reason to be smarter than the other. Fable 5 will consistently draw more connections, come up with better ideas, etc.

Examples of strong reasons for one to be smarter than the other:

1) Fable has millions of subjective years doing GPU kernel optimization, I've barely done any throughout my life.

2) On the other hand, I have private information and experience with my x-risk comms job that is completely absent from AI training data and its consequences are at least a few non-trivial logical steps deep, so they won't be immediately grokked within the context window. Together with my colleagues and wider social network, I've had plenty of time to digest this information.

[Nux @mherlihy0816]

Someone asked if I was ready for the fall and it took me a second to realize they meant autumn and not the total collapse of society.

[Alex Amadori @testdrivenzen]

@Polymarket https://t.co/S0ZTQ0n32Q

[Connor Leahy @NPCollapse]

We agree with President Trump: America must stay ahead of China in building AI for economic competitiveness and defense. But superintelligent AI is something else entirely: not a tool, but swarms of fully autonomous AIs that we cannot control, powerful enough to outcompete humanity.

Developed anywhere, superintelligent AI threatens the life of every American. We are excited to see the Ban Artificial Superintelligence Act open the discussion in Congress about how the U.S. should address this threat: ban superintelligent AI at home, and lead the effort to prevent its development anywhere else.

Now that this bill has kicked off the discussion, we encourage Congress to focus on preventing superintelligent AI. Rather than a pause on all advanced AI, we recommend targeting the projects pursuing superintelligent AI as we speak and putting American security at risk.

https://t.co/Wpk8LOzDSi