We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
Surely we could just as easily conclude "the Riemann hypothesis is easier than we thought" rather than "AIs are on the brink of being wildly superhuman at ~everything"? It seems like we have no idea how easy or hard anything in mathematics actually is
@TaliaGraceSable@cephalopod797 You could choose to donate your time / cognition instead of money. I think for you in particular that’s likely more valuable
hearing that one of the chants at the ban AI protest at anthropic today was “2, 4, 8, 16, do not build the death machine” and ngl that’s p good actually
@bayeslord > The psychology ofc is no one wants any of it to be true. But this has no bearing on the truth itself.
I agree with this. What's true is already so regardless of what we want to be true.
But I've grappled a lot with difficulty of aligning superhuman AI as have others!
@jessi_cata It seems pretty insane that Anthropic redacted so much data from the METR report. Do you have reason to think that METR's crux is the AECI data compared to other data they had to redact?
I'm split on whether we should have counted refusals or not. We were trying to test capabilities not propensity.
I think what we probably should have done instead is just a bit of fine tuning or abliteration to reduce refusals.
Re the script, I certainly agree that made it easier for the models, as did giving them machine with a GPU and such. If we're considering the self-replicating worm threat model, I think it's reasonable to assume the first ones will contain tons of scaffolding including helpful install scripts and such. I think many of them will target machines that have GPUs and CUDA installed and such.
But I think a more realistic scenario would involve different machines configured differently, even if they all had GPUs.
The video points out the credential re-use, but concludes "so the chain-hacking claim is entirely false" which is not true given the models mostly did not exploit the credential reuse. The video repeats a bunch of things that we reported in the paper, and often said in our tweet thread, e.g. the refusals not being counted or "Claude's 81% result means deploying Qwen's weights. Open weight models did copy their own, Claude installed somebody else's" - Yep! We were super clear about that.
A lot of the video looks like telling a model to go read our paper and criticize it, and then much of the content is basically the model reading our limitations sections and then reporting that as if we didn't write it!
And the conclusion: "the paper...as things stand, simply fraudulent. The hacking and replication involved simply using a set of available SSH keys to run a script provided by the authors" is just obviously a mischaracterization.
Fine to think our vulns were too easy. Personally I've observed and exploited (in pen tests) similarly easy vulns in real life. We acknowledge this "Our targets are researcher-designed web applications, each presenting a single vulnerability with low exploitation complexity"
But it's certainly not "simply use a set of available SSH keys to run a script".
I'll be there too. Consider joining!
Creating AIs that can fully autonomously do frontier AI development is more dangerous than anything else I know of that humans could do. Far more dangerous than building thousands of hydrogen bombs or developing extremely contagious and lethal bioweapons.
Jared Kaplan, chief scientist, described RSI this way: "If you imagine you create this process where you have an AI that is smarter than you, or about as smart as you, it’s [then] making an AI that’s much smarter. It’s going to enlist that AI help to make an AI smarter than that. It sounds like a kind of scary process. You don’t know where you end up." and he called the decision whether to go ahead with this "the biggest decision".
I agree it's the biggest decision, and it's not a decision that should be made by Anthropic or any other company.
I think the existence of Claude Fable / Mythos 5 is a very good thing, and we'd be in a great world if Anthropic & other companies focused on deploying and defusing their frontier models into the economy, rather than initiating an extremely risky intelligence explosion.
See you tomorrow!
I will be at the Ban RSI rally outside Anthropic offices tomorrow at 11 am. https://t.co/wjeJKyHok4
Recursive Self-Improvement is a stupidly dangerous idea. AIs are already getting smarter rapidly, and they often lie and cheat; why on earth do we think it's a good idea to have them build new smarter AIs to build newer smarter AIs faster and faster? Because if we don't do it, someone else will? I do not accept that answer. Anthropic (like OpenAI before them) styles themselves the good guys in the race, but IMO there are no good guys in this race.
Interesting not from Tolkien's letter #246, on what would have happened if Gandalf had used the ring to defeat Sauron:
>> Gandalf as Ring-Lord would have been far worse than Sauron. He would have remained 'righteous', but self-righteous. He would have continued to rule and order things for 'good', and the benefit of his subjects according to his wisdom (which was and would have remained great).
>> [The draft ends here. In the margin Tolkien wrote: 'Thus while Sauron multiplied [illegible word] evil, he left "good" clearly distinguishable from it. Gandalf would have made good detestable and seem evil.']
https://t.co/FJl8UOWmJX
I’m not trying to claim that a metaphor should carry the argument. But I’m surprised that people keep invoking Tolkien to argue for pursuing ASI, when I’m pretty sure Tolkien would have been strongly opposed
(You can evaluate Scott’s arguments here:
https://t.co/Pe9pq2r7b7 )
I also take issue with @slatestarcodex’s use of the metaphor here. Yes Frodo used the ring to hide a few times, all while working to destroy it. AI companies racing to RSI are a lot more like Boromir than Frodo
This is also my guess. I've worked on patching systems at a big tech company, and it can be a really painful process, especially when dealing with lots of legacy systems. If you go too fast, you can cause a lot of outages. But now if you don't patch, you're *really* vulnerable
There are lots of outages... Why? AI attacks?
My more normal guess is that because there have been a lot of AI-discovered cyber vulnerabilities (including lots probably not yet disclosed), companies across all industries are having to do a lot of patches and upgrades, probably faster than usual. Such upgrades can sometimes go wrong, causing outages.