@willmcgugan Just want to take this opportunity to say that Textual is a joy to develop with, and that it’s been the prefect tool for a few of my projects. Thank you!
I think it's fine to make vague value statements that are not yet perfectly fleshed out laws. People have been doing this from the abolitionist petitions of the 1700s - many of which were just a few paragraphs about how slavery was bad, with zero plan to uproot an institution that dominated the economy of half the country - all the way to the Harpers letter on free speech a few years ago, which also just said we should "preserve the possiblity of good-faith disagreement" with no explanation as to how. These were good and worked well. Once there was common knowledge that people cared about these issues, some people responded to the implied social norm directly, and other people at organizations and governments were inspired to come up with specific plans and laws.
I don't think the statements about scientific consensus around safety necessarily involve a CERN-like effort. I imagine this as continued safety research of the type we have now - for example, mechanistic interpretability. When GPT-5 doesn't blackmail users in contrived situations, when Chris Olah announces he completely understands Claude's moral decision-making process, when agents can't be prompt-hijacked by trivial exploits, when AIs stop lying and hallucinating, when we go a year or two without some new bizarre memetic parasite - then I can imagine there being "scientific consensus" that the time has come to try scaling up.
But if there is a CERN-like effort, how is that worse than what we have now? You seem very concerned about "the people who possess, in other words and in the final analysis, the right to kill you or confiscate your property if you do not listen to them, newly empowered with the most powerful technology ever conceived" - as if the Chief Scientist of CERN were the only person greedy enough to consider taking the Ring for themselves. Is the leader of CERN - currently a nice Italian woman with a background in classical music - really that much scarier than Sam or Elon? And do you really think that the US government - the same organization which couldn't even let Facebook develop a social media algorithm without finding a way to turn it into a tool of censorship and control - would just not notice when OpenAI or XAI had a 98% complete world-domination machine ready to go, entirely in American territory, in a company that the President could destroy or confiscate with a stroke of the pen? I kind of feel like "I don't trust governments with superintelligence, so let OpenAI or XAI build it" is the same kind of argument as "I don't trust governments with fighter jets, so let Lockheed Martin or Boeing build them".
If you really don't trust anyone with the Ring, then I think you and the signatories here are in agreement. The only person I've heard with a clear plan - Eliezer - doesn't mention a CERN-like entity. He says don't build it at all. Monitor compute, IQ-enhance humans, work on some technologies like mechanistic interpretability, then let the IQ-enhanced humans figure out when the mechanistic interpretability had advanced enough that it's safe to lift the ban. As to how to lift the ban - whether this involves CERN, or Elon Musk, or a thousand carefully-nurtured open source projects, or something else - I think we leave that one to the IQ-enhanced humans.
Vague statements like this, which fundamentally cannot be operationalized in policy but feel nice to sign, are counterproductive and silly. Just as they were two or so years ago, when we went through another cycle of nebulous AI-statement-signing.
Let’s set aside the total lack of a definition of “superintelligence.” I’ll even grant the statement drafters that we all arrive on a mutually agreeable definition.
Then assume we write that definition into a law, which says “no superintelligence until it’s proven safe.”
How do we enforce this law?
Now comes the fine print—the stuff left unsaid in the statement, the stuff the statement drafters probably did not much discuss with the many signatories who lent their names and reputations to this endeavor.
How do you prove superintelligence will be safe without building it? How do you prove a plane is flightworthy without flying it?
You can’t. So, the logic would go, we will need a sanctioned venue and institution for superintelligence development, where we will experiment with the technology until it is “proven safe” (who decides this, by the way, and what happens after it is “proven safe”?)
This institution would need to be funded somehow by all governments with similar prohibitions (which the statement drafters, though probably not all signatories, would likely argue needs to include every country on Earth, including US adversaries).
A global governance body whose purpose is to build the thing the statement drafters have told us is so dangerous, partially because of the power it could confer on those who control it. A consortium of governments which, if successful, would exercise unilateral control over how to wield this technology—and against whom to wield it.
The same people who uniquely possess militaries, police, and a monopoly on legitimate violence. The same people who possess, in other words and in the final analysis, the right to kill you or confiscate your property if you do not listen to them, newly empowered with the most powerful technology ever conceived.
Does that sound “safe” to you?
This sounds to me like the worst possible way to build “superintelligence.”
I reject all efforts to centralize power in this way. And I reject blobby statements with no path to productive realization in policy.
Richard Feynman
“Science is the belief in the ignorance of experts. When someone says ‘science teaches such and such’, he is using the word incorrectly. Science doesn’t teach it; experience teaches it”
Dr. Jane Goodall filmed an interview in March 2025 with the understanding it would only be released after her death. This is her final message from it.
We beat Nvidia’s cuBLAS kernels on B200s in 170 LOC.
Using zero CUDA. Just pure Mojo.
Here’s exactly how we went from 1% to 106% of Nvidia benchmark perf from scratch (with code) 👇🧵
Ben Franklin appreciation post
He was simply S-tier at every endeavor:
- Invented bifocals, swim fins, and the lightning rod
- Diplomat, statesman, charted the Gulf Stream
- International playboy; more like founding daddy amirite
- Created the first public library, the first volunteer fire department, and UPenn
Truly contained multitudes, the Yamaha Corporation of humans
BREAKING: MIT just completed the first brain scan study of ChatGPT users & the results are terrifying.
Turns out, AI isn't making us more productive. It's making us cognitively bankrupt.
Here's what 4 months of data revealed:
(hint: we've been measuring productivity all wrong)