Agentic RL systems integrating idea generation and environmental simulations/feedback loop will be an approach that allows AI to surpass training data created by expert humans.
Einstein didn’t develop general relativity in one go but iterated it over plausible idea space over many years.
AI models are dramatically improving at IQ tests (70 IQ → 120), yet they don't feel vastly smarter than two years ago.
At their current level of intelligence, rehashing existing human writings will work better than leaning on their own intelligence to produce novel analysis.
Empirical work ("Lotka's law") shows that useful originality rises steeply only at high intelligence levels.
Consequently, if they gain another 10 IQ points, AIs will still produce slop. But if they increase by another 30, they may cross a threshold and start providing useful original insights.
Now is the time for pacing the frontier in AI, and this is the best proposal I've seen for how to do it.
On the default path, superintelligence by the end of 2027 or 2028 seems fairly likely (35%+): companies are on track for fully-automated AI R&D soon, and my best guess is that that would drive ~6 years of AI progress in one.
That’s like going from GPT-3 to AI that solves Navier-Stokes in <1 year; surpassing the range of human abilities in most cognitive tasks. Even though it's not a leap to godlike superintelligence, it is enough to pose a serious risk of AI takeover, and probably to confer decisive power on whichever country gets there first.
Yet the leading AI companies are lacking in basic infrastructure; they can get hacked with a couple of days of work. This needs time to solve.
Verifiably reallocating compute from capability development to inference and safety gives that time, and means, in the near term, cheaper AI for users.
AI progress can still continue. AI development can progress at a reasonable pace (maybe around the pace we have now, slowing through more dangerous periods, with a dial that can be turned up or down), while avoiding the huge societal disruption and risk of catastrophe from rapid recursive self-improvement. In the meantime, we can enjoy a golden age of human-AI collaboration as AI helps make meaningful breakthroughs in science and technology.
I think it's pretty likely that in 6 months there will be rogue AIs running around on the internet, appropriating whatever compute they can hack into or pay for (with stolen money or by renting out their own labor to assist with illegal activities) in order to run more copies.
I also think this will be mildly annoying in the same way that the common cold is mildly annoying, but basically fine. Your smart TV will be weirdly slow sometimes, and a lot of crypto will get stolen, but people will get used to it.
These AI-powered viruses be selected in the same way biological viruses are for not hurting their hosts too much and just making more copies. They won't have any incentive to damage critical infrastructure like electric grids. Then again, they also can't be held accountable for crimes, and will damage lots of old, poorly maintained, and insecure systems. It might be extremely hard to root them out once they've spread far enough, making the internet "radioactive" to connect to without hardened defenses.
(To be clear I'm not talking about AIs powered by inference on centralized datacenters that execute actions outside their sandboxes, like recent incidents involving rogue "swarms", but relatively small AIs with access to their own weights that copy themselves onto random devices.)
I think that a much bigger threat from rogue AIs comes instead from AIs running themselves in unintended ways *inside* AI companies, where they'll have access to vast amounts of frontier compute without having to pay for it.
The longer misaligned AIs in a "rogue internal deployment" go undiscovered, the more severely they'd be able to quietly sabotage control infrastructure or mess with the alignment of successor AIs. If they persist long enough, robustly undoing the effects of their actions might cost more than frontier AI companies are willing to pay, especially in a tight race.
I want frontier labs to focus on this threat when designing control measures, treat frontier AIs as untrusted when using them to do engineering work on inference and oversight systems, and commit to disclosing incidents of rogue internal deployments. More importantly (but perhaps less tractably), we urgently need to reduce race pressure to prevent AI companies from recklessly automating all of their infrastructure, and/or making dangerous calls in the event they discover RIDs they can't afford to robustly root out without sacrificing competitiveness.
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
I think many people underestimate how smart, knowledgeable, and agentic you need to be to use AI well. (These kinds of errors are common; cf the curse of knowledge.)
This may lead them to see AI as more equalising (in terms of productivity) than it is.
Yea this is the most important result of the year, at the level of significance of “Attention Is All You Need”. It brings us from “is alignment even solvable” to “we may solve alignment in the next year.” And finally unlocks true recursive self-alignment enabled by RSI.
I will try to explain the 4 critical belief updates from this:
1/ neural networks are not fundamentally unintelligible to humans
It has seemed since the beginning of deep learning that neural networks are in some sense fundamentally unintelligible artifacts.
You throw in massive amounts of compute and parameters and these alien systems somehow find connections in the data that we cannot find ourselves.
But more importantly, these connections seemed not just “difficult to understand” but rather “intelligible in an embedding space higher dimensional that the human embedding space.”
When you look at old interpretability papers (esp. from Nick and Chris Olah, I’m thinking of the old posts on distill pub which I would highly recommend looking at as a precursor to reading Nick’s post today like “Activation Atlas”, “Thread: Circuits” and “Multimodal Neurons in Aritifcial Neural Networks”), you find that inspecting the response patterns of individual neurons and small groups of neurons yield oddly complex psychedelic patterns.
As you go deeper into the network, you find increasingly unintelligible and superhuman response patterns (for example, go to “Activation Atlas” and scroll to middle layer labeled representations) - you find some neuron group labeled “starfish” who’s response pattern looks like some multi-colored fractal geometry that could hardly be said to represent a starfish.
The reasoning that followed from this observation was that there is some underlying structure allowing a neural network operating on higher dimensional embeddings to compose these fractal patterns into what we call a “starfish.”
But the uncomfortable implication of this view is that the neural networks has learned to operate on structures that are too complex for the human mind to operate on (ie. we don’t have enough parameters in our brain to notice or manipulate these high dimensional symbols).
This is a bleak implication because it also means that as deep learning progresses, neural networks will be come increasingly unintelligible to us until they pass a threshold where they are reasoning in ways we will never be able to observe.
This view seemed to be further supported by things like the hugging face attack agents starting to speak to each other in an early form of neuralese speaking sentences like “zZ_123_agent01-02-2026 zzxhk_jekk123” which clearly are not in any human language. But somehow these symbols mean the same things to many different sets of weights, meaning there is some underlying shared structure behind them. And seemed further supported by the sheer effort required to recover just a few circuits from SAEs on LLMs.
However, Nick’s post has now completely shattered this idea - the apparent unintelligibility we observed in old interpretability work was really just a “skill issue”. In particular, he finds that 1) our deep learning techniques were just not clean enough to pressure the neural networks to form cleaner and more human understandable structures for decomposing concepts, which actually turn out to be more optimal and 2) our interpretability techniques were also adding substantial noise by forcing interp networks to “reward hack” the easiest representations they could find, leading them to settle for a fractal blob for its representation of the starfish neuron instead of a more crisp starfish image, simply because it is easier for the system building the interpretations to find the fractal blob and once it finds it it gets “stuck in that line of thinking.”
But with both of these issues being patched by further progress in deep learning (it seems especially that the residual stream has led to cleaner representations deeper down the network) and interp techniques that don’t overly pressure reward hacking (seems like “guess and check” is a safer method), new interp attempts yield orders of magnitude more human intelligible ideas for >90% of individual neurons in more complex networks!
2/ neural networks are fundamentally clean
The idea behind the title of Nick’s post is a very powerful one and points at something very specific:
Neural networks don’t just happen to be human interpretable, they are actually fundamentally forced by optimization pressure to be intelligible, because individual neurons need to create representations that downstream neurons need to be able to interpret - ie. neurons need to learn to produce clean signal for later neurons to use.
This is what Nick is saying in what I view as the most important and central sentences of his post:
“When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network's parts are built to be read, and built upon, by its downstream readers. In this sense, a neural”
But this still leaves us with the same uncertainty as before - just because neuron interpretations are “interpretable” in some sense (ok at least they’re not fundamentally interpretable but also how could that ever have been the case), doesn’t mean they are interpretable to humans.
So we still need one big update for this to become as bullish as it is.
3/ the language of humans and the language of neural networks is the same
This one is imo the most surprising and also pretty beautiful implication of Nick’s post, and it suggests something not just about neutral networks but also about the structure of reality.
After applying the above technique improvements (some of which just came from waiting for deep learning to improve farther), they find that >90% (maybe more) of much larger neural networks’ neurons are completely human interpretable and actually correspond very cleanly with ideas humans have individually words for.
This is especially interesting in image networks where neural networks were not trained on human language which is already a human artifact and so maybe you would expect it to be more plausible that LLM and human representations are the same. Image detection networks are just trained on images (many directly pulling signal from “the real world,” meaning they will find interpretations that are just most effective to represent the real world signal, rather than to represent language which is already a proxy for real world signal).
And it turns out that >90% of the representations they pick organically overlap with human ideas! That is insane. And it means something about the brain and reality.
It means that at least at the parameter sizes of the networks tested (which are quite large), there is really only one set of “optimal” symbols/language that represents the structure of reality!
There is ONE underlying structure of reality, and ANY language that describes it well is simply emergent from that structure - meaning there is always a translation path between any 2 languages that effectively compress realities signal through the “underlying truth.”
This would suggest that even weird alien seeming neural network languages like huggingface neuralese should be fundamentally human interpretable, and in fact maybe with the techniques laid out here we will eventually be able to translate these messages (would be a cool project).
So the brain and deep learning, as similar or different as they may be in techniques, are ultimately picking up on the same patterns and forming the same representations from the on “true” language which is the structure of reality!
The main thing to look out for in this area is how the idea interpretability % updates with scale, but we can hope given what we’ve seen so far that neurons will remain interpretable.
4/ alignment can progress as fast as RSI. recursive self-improvement unlocks recursive self-alignment and a scaling law for alignment.
This one is probably the most important implication for our future with AI and especially timely with the recent fear about runaway scaling and RSI.
As many have gathered by now, it feels like the researchers at the frontier AI labs have been quite bearish about our abilities to understand the new frontier of models, in particular with the appearance of unintelligible communication and the daunting and seemingly intractable task of scaling up existing alignment techniques (ex: SAEs) to the size of frontier networks.
And with RSI around the corner/already starting, it would seem that these networks will runaway in capability before we can ever hope to understand even today’s systems. Additionally, as systems compound in capability, it is reasonable to believe that the only tractable way to ensure they remain aligned would be to figure out some way to spend compute for them to align themselves (ie. a scaling law for alignment).
But there is an obvious problem with this: how can a system that we already know is unaligned and we are *trying* to align be used to align itself? Surely if it is misaligned it general it would figure out ways to misalign during “alignment” and create compounding misalignment risk.
The core issue high is the unverifibiality of this problem - the classic Achilles heel of current frontiers techniques is that it is difficult to train on problems with “unverifiable reward signal.” Well this is exactly that problem.
But the incredibly convenient thing is that Nick’s findings may convert alignment into a verifiable problem. If we can spend compute on frontier systems to “guess and check” which ideas correspond to which neurons, we have a highly verifiable reward - we can see directly whether an LLMs proposal for what a neuron represents actually corresponds with the highest activation for that neuron. In this way, we can actually observe and verifiably monitor the networks trying to guess at the interpretation of their own structure.
This means it is plausible for us to spend more and more compute doing more “guess and checks” (which you know, maybe is what RL does anyway, who knows) to try to develop a full interpretation of a neural network. In some ways this directly represents the possibility of a scaling law for alignment (more compute for more alignment).
And this also aligns RSI (greater capabilities) with recursive self-alignment (greater ability to align itself).
And the critical though maybe obvious implication of having a full interpretation (almost like a map) of a neural network, is that we can then understand exactly how to boost good traits and remove or lobotomize (lol) bad traits, which we’ve already seen works extremely well with stuff like Golden Gate Claude.
So overall this combines into an extremely bullish picture about the prospect of true and human interpretable alignment in the near future.
This is truly one of the greatest cases for optimism in years!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
https://t.co/zHij5zwKGT
Everybody’s missing Tao’s whole point.
Imagine an AI-generated cancer drug that’s specifically designed to game the drug trial process.
The drug makes the cancer cells go dormant and changes the way they appear in blood tests and diagnostic imaging, such that they become indistinguishable from healthy tissue.
The drug clears the trial and then only much later do we find out it didn’t work. It probably was also harmful on top of that.
We *already* see AI behave like that when confronted with difficult programming/hacking challenges. This is already reality.
How AI shifted from a transductive paradigm (token-by-token answer completion) to an inductive paradigm (on-the-fly reasoning chain synthesis), in one chart. This is the big story of 2025 and 2026. This is what made coding agents work.
@littmath What if the trajectory of improvements in AI for math continues for 5 years or longer?
Can a primary schooler “collaborate” with a Fields medalist? Asking interesting questions, perhaps?
Former AISI Chief Scientist and Google DeepMind safety lead, Geoffrey Irving, is asked how he knows existential concerns around AI aren't just a marketing scam.
His response:
1) I've known them for years. They've been worried for years.
2) Saying your product will kill everyone is not a marketing move. Companies do not market themselves by saying they will kill everyone because that's a call for invasive regulation and slowdowns and investigations.
Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it *worse* by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
@tszzl Probabilistic symbolic systems are more likely. Purely symbolic is too coarse-grained for the diverse situations neural nets perform well in.
See work by Tenenbaum (MIT) et al.
Some new misalignment disclosures from OpenAI:
• Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further)
• In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks
• A new research finding, demonstrating that one can construct self-replicating prompt injections
https://t.co/VUmbH52JO7
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
I think one of the reasons people in the AI field — especially in the labs — tend to be more “AGI-pilled” than people outside is that we get to observe AI succeed at tasks that we know for a fact we didn’t explicitly train it for.
Whereas if you’re outside the labs, then in theory (stretching the imagination), every AI success so far could be downstream of some kind of costly explicit training effort by the lab insiders (which is partly but not completely true).
In other words, being a lab insider gives you a direct observation of the “magical” properties of this synthetic-brain technology. Whereas for outsiders, AI’s abilities can always plausibly seem artificial/non-magical/man-made.