Anthropic engineer said something that completely changes how you think about AI agents:
"Don't prompt Claude. Build a system that prompts itself."
In just 26 minutes, he walks through exactly how that works from the core idea to building the entire system step by step.
Most people are still manually prompting AI for every task.
The next generation of builders is designing systems that can plan, execute, evaluate, and improve on their own.
Watch the talk first, then save the guide below you'll look at AI agents very differently afterward.
Of all the 38-word messages I'd choose for 100 million people to see, Jacob's below would be at the top. It is simply correct. The companies are racing towards something breathtakingly dangerous. They are gambling with our lives. They know it and are afraid. They feel trapped in a race and don't think they can stop.
Governments are horrifyingly far behind on this. There are no adults in the room handling it. I don't know what prevents this, but mass awareness, and sustained public pressure on our governments to act, will surely help.
[Writing this in a personal capacity, not on behalf of my employer (Anthropic).]
Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:
1. AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.
2. Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.
3. Unlike traditional software, we can’t “program” AIs to behave how we’d like. AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.
4. We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them. Insofar as there is a plan, it’s to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs.
5. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. That was the intent of this open letter (which I signed): https://t.co/TZOm3LfptY
I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.