Nathan Heath

Founder & CEO @syntonyresearch | 🤖🔴 Red Teamer @OpenAI @anthropicAI | 🗺️ Fellow @trumanproject @FP4America @bluedotimpact | RT 🚫 Endorsements

[Sam Altman @sama]

I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue.

[Pope Leo XIV @Pontifex]

In this era of artificial intelligence, it is becoming urgent to distinguish human art from what machines produce. There is an ontological difference, even before an aesthetic one, between art and what a machine can generate through statistical calculation based on millions of images created by others. Algorithms lack the spark of humanity. For this reason, the Church wishes to renew an alliance with artists and cultural institutions to safeguard our humanity.

[Nathan Heath @nathanheath92]

@AnthropicAI is looking like a cult more by the minute, with an extraordinary influence and dangerous ideological bent that really needs to be checked and @OpenAI is little better for other obvious reasons…

[David Decosimo @DavidDecosimo]

This is deeply disturbing:
Anthropic secretly invited religious leaders to SF to advise them on AI safety, had them sign NDAs, but then primarily tried to convince them Claude has a soul & moral standing, all while wining & dining them to the max. 1/ https://t.co/4Z7Yk9rwk2

[Dave Peasant, Aspiring Lorax @aspiringpeasant]

THIS IS A PARKING LOT IN ARLINGTON

[Nathan Heath @nathanheath92]

Actually one of the single best things to come out of the state of Florida. #respect

[Joe Cohen @CohenSite]

The Florida Attorney General has filed a temporary injunction against OpenAI, and it, unfortunately, slaps

(h/t @MelMitchell1)

[International Cyber Digest @IntCyberDigest]

The agents escaped the sandbox again

Meanwhile the agents' owners:

[Jacob @Burgess_33]

By far the best genre of slop so far 🤣

[Anthropic @AnthropicAI]

Claude has discovered a previously unknown enzyme system hidden in the DNA of bacteriophages. Beside the enzyme’s gene sits a long array of repeating DNA—a structure that looks somewhat similar to CRISPR.

We don’t yet understand what this system does, but only a handful of known systems share its features, and all of them are able to cut, copy, and paste DNA. Historically, the discovery of such programmable systems has helped revolutionize medicine. CRISPR, for instance, is now the foundation of genetic medicines. But it will take much more work to learn what this system does, and whether it can be put to similar use.

Read more: https://t.co/RuEosScSMb

[Jak Casey @jdpcasey]

@claudeai I'm sat here trying to decipher "made the interval closed on the wrong boundary" after them telling me it can write naturally again 😭

[Pedro Domingos @pmddomingos]

Exactly. Hold OpenAI accountable and don’t buy its “swarm of rogue agents” stories.

[Gary Marcus @GaryMarcus]

If EA’s and Doomers were serious about making AI safe, they would bone up on cybersecurity and use every tool available to keep AI under control, instead of siding on X with the very people building reckless systems.

The fact that they simply won’t engage in the reality of basic cybersecurity lapses in favor of extremely vague stories about alignment—with no plausible suggestion at all about how to solve that—tells you everything you need to know.

[Andrew Ng @AndrewYNg]

The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.

I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so.

First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world.

The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability.

Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended.

I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.

Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.)

Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.

Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building.

[Original text (with links): https://t.co/jni2tWazAH ]

[Andreas Kirsch 🇺🇦 @BlackHC]

I think debates that argue that shrimp or insect welfare matters more than human welfare in aggregate suffer from comparing incomparable things and then feeling clever about pointing at some numbers and saying we shouldn't trust our institutions

Okay here is my turn:

Why do we not care about individual cells bc there are the building block that is alive. Cells experience cell death via signaling mechanism, too, and they also try to avoid it

When it comes to such a comparison, human cells vs insect cells, humans are not that far behind in total

Maybe only 5x less than all insects together (going from approx biomass), so actually humans do matter a lot more compared to any given specific insect species!

Oh and against shrimp: we win by 5-20x 😇

[Stanislav Fort @stanislavfort]

Morality isn't solved, and any model of it is likely only a limited approximation. Newton's gravity fails when you try to predict things like black holes, outside its original regime of applicability. In the same way, naive mathematical theories of utilitarianism lead to crazy conclusions out-of-distribution.

I think the sane reaction is not to "bite the bullet" but rather to tap out and realize that the underlying theory might just not generalize all the way to the regime you're trying to use it at.

It is not a virtue of rationality to take things to their extremes. It's rather a failure to appreciate the limitation of the underlying models and their premature mathematization that lends them false air of rigor.

[Max Rose @MaxRose4NY]

We can chew gum and walk at the same time.

1) pass AI federal legislation asap that includes safety guard rails

2) arrest every ai executive who is knowingly building a product that has the capability to end humanity

[Neel Nanda @NeelNanda5]

If you're curious who METR, the "woke wizards of AI", actually are, this is a great post. If someone is threatened enough to fund a smear campaign against them, you know they must be good at their jobs!

[Dr Heidy Khlaaf (هايدي خلاف) @HeidyKhlaaf]

@jackclarkSF What situation? That Anthropic couldn't configure sandboxes and deferred to natural language as a way restrict agent access to the internet? "Stochastic parrots" is a convenient straw-man to not engage with security arguments against false narrative of agents going "rogue"?

[Matthew Yglesias @mattyglesias]

Guy who thinks AI will wipe out humanity and that’s good because of the improvement in shrimp welfare.

[Michael Tontchev @MichaelTontchev]

If you plan to share this article - don't. It's plain wrong.

"The attack in no way represents some sort of rebellion by the AI models. . . . In fact, they did exactly what they were told to do"

False. OpenAI did not tell the models to hack HuggingFace. If you tell your robot to make sure to feed your dog, it's absolutely misaligned if it decides to feed your cat to your dog for lack of remaining dog food.

"One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly.’ There was a live route to the internet"

Agents must be robust to both underspecification and misconfigured environments. If they are not, they are misaligned. If you tell the agent to maximize paperclips, it should not turn the earth into miniature paperclips. This is basic stuff. "But there was a tool that would allow it to do it" does not mean the agent should use the tool! If the agent cannot complete the task, it should make whatever partial progress is *reasonable*, or else just stop. Again, super basic stuff.

The only way to accept the (obviously hastily AI-written) takes in the article if you contort words to mean what they clearly do not - and contradict the safety literature spanning decades.

If you build an agent to pursue goals, you are *not* looking to train it to pursue goals at literally all costs. If it does, that's exactly what safety people have been calling misaligned for decades!

[Claire Lehmann @clairlemon]

The fact that EAs are surprised that a moral framework most of humanity doesn't share is being criticised--at the exact moment AI companies are becoming some of the most powerful entities in the world--doesn't reflect very highly on their ability to model the world accurately.