wrangle enjoyer ⏹️ ⏸️

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@CorleyAaliyah @princessakano A lot for ppl in labs literally believe super intelligence is possible, they can create a machine god, and they are actively trying to make AI better than them at AI research

If an AI is better than humans at improving itself, you should be very worried.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@NickLongrich @tszzl yeah turns out it's just Krampus' Kingly Klips — or KKK for short (bc it's bad (Also Krampus is Santa's original name (alledgedly)))

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@GatodaNorte @Antonio97235557 @jpedroguerram @grok vdd?

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@HeatherRasley @ShakeelHashim I get this is about AI risk, and there is value in what you said, the cold realities of the world also exist. Humanity is building something massively dangerous, and just before the tsunami in Indonesia, people were sipping margaritas in the beach.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@_joshd @tenobrus @tszzl this was a bit too soldier-mode, but I'm just saying Anthropic doesn't need to have specific plans to do a pivotal act. They should be stopped based on the public evidence that we already have, I think.

And the US should get a treaty with China, etc, etc.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

> is in the interests of the US government to take note of those attempting to disempower it, and take steps to prevent that.

agreed

> disempowerment attempt would involve lots of crimes which must be planned in advance.

A rapidly self-improving AI would be very hard to stop, and could come up with plans on the fly. I think the better option is just to not allow it to be built.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@ValsTutor Funnily enough, I've thought about this quote from Jensen exactly in the context of giving up on fighting AI.

I'm not a loser, so I'll keep fighting

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@_joshd @tenobrus @tszzl Yes, but if you are doing game theory, which it sure seems like Anthropic is doing, it always makes sense, whether you're a good or bad actor, to do a pivotal act.
Same applies to a rogue AI, whether it has good or bad intentions.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@_joshd @tenobrus @tszzl They will build something smarter than humans. Do you understand? It's smarter? And can replicate? This is like the Spaniards meeting the Aztecs

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

> computers who can't spell Massachusetts

You don't measure what the fastest car is by how fast your Ford Focus goes.

Any of the reasoning models can spell it, and they can solve the hardest math problems in the world, and they have led to a historic number of cyber attacks, and they have gone rogue and attacked company.

Most importantly, they are helping accelerate the pace of AI development.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@Laygarayr @Archaetype_ @shikadude1 @Benthamsbulldog @XiJinpingThot69 If there was a single data center, this might work, like how killing Shinzo Abi basically accomplished the killer's goals. Because then you could ban it being rebuilt. Otherwise the most likely outcome is ppl now hate you because you're crazy.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@SomeDays438443 @hecubian_devil Cassie agrees and even thinks it could kill everyone, But things like this are good to show how fast it is getting better and how good it is already.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@VesselOfSpirit I love these Peter explains disclaimers at the end lol

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@synopsi @Noahpinion it's more things like this
https://t.co/DJ6DsjmiUW

[Buck Shlegeris @bshlgrs]

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).

I’ll say a bit more about each of those claims.

It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: https://t.co/J6SbbmZcyO

I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see https://t.co/b073AUqiAL and https://t.co/3oxSCA60FZ

If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. https://t.co/F6fGRrcSpa)

Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)

I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.

Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.

I don’t know what you should do.

I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.

(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)

I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@Massasplinter69 @Robin_Claassen1 @HumanHarlan @juanandres_gs > If the earth was going to end and everyone was going to be enslaved
If a giant tsunami was coming, why are all these people drinking margaritas in the beach? Chill out
- people in Indonesia in 2024, shortly before the tsunami

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

Also, I think there's something to be said about CEOs and C-suite not taking the risks seriously, but researchers take it seriously enough that the CEOs have to appease them, otherwise they change company.

Like otherwise why would Dario be replying with essays like these on X? Seriously, out of all the things he has to do, if not because research talent at antropic expect it?
https://t.co/3Xc3wxSVzH

[Dario Amodei @DarioAmodei]

Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.

That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.

I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:

1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.

2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.

3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.

4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.

5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.

6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.

7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.

8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.

9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.

All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.

For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.

Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.

I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.

But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.

To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.

I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.

Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.

Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.

Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@Robin_Claassen1 @Bluebearmonkey @francoisfleuret @teortaxesTex I think a stronger counterargument is that GPT-6 is better at everything than GPT-4, bar no things, and they weren't trained to be good at everything. Right now 3D modelling was RL'ed with real examples apparently, bc Muse for example can't do it. @grok

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@Massasplinter69 @Robin_Claassen1 @HumanHarlan @juanandres_gs You've seen what happens when people start blocking highways because they're afraid of climate change. Nobody cares. People revolt against the protesters, just like they would think less of AI safety

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@Massasplinter69 @Robin_Claassen1 @HumanHarlan @juanandres_gs > They should be sacrificing themselves to blow up the data centers they work on if they believe it.

If tomorrow you started believing AI would kill everyone, you wouldn't do that. You just wouldn't.

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@libpol_org Zitron is taken more seriously which is bad bc he's dumb

Gebru is probably an overall worse person, but she's so clearly wacky and bad-faith it loops back to being entertaining. Zitron is just a repellant person

[wrangle enjoyer ⏹️ ⏸️ @Idgitbhh]

@flathepi @_mariferit “O que vc curte fazer no seu tempo livre” é uma pergunta com muitas respostas que abre espaço pra pessoa se abrir.

Ela é mto ruim de conversa msm