@JacobHHilton I think I have each neuron’s main job, and Claude rebuilt 16-neuron model by hand, with no training: 96.0% vs 95.3%. Top: ARC’s weights. Bottom: hand-crafted (same main function). Off to the vet, post soon, more juicy stuff
Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied!
We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵
if ai labs can spend $10M+ of compute on solving math problems, they should do the same for closed-ended mechanistic interpretability problems.
negative results would also be informative.
@JacobHHilton I think I have each neuron’s main job, and Claude rebuilt 16-neuron model by hand, with no training: 96.0% vs 95.3%. Top: ARC’s weights. Bottom: hand-crafted (same main function). Off to the vet, post soon, more juicy stuff
@karpathy I’ve started playing around with explainer puzzle games in HTML. One shotted Opus 5.5 games are already quite nice. I expect to become addicted, tickles the same parts as chess puzzles and go tsumego.
i have many disagreements with effective altruism, even when i was a rationalist i never thought of myself as an EA, but the recent spate of anti-EA nonsense is so blatantly obviously in bad faith i am compelled to do a "nobody gets to pick on my cousin except me" here
EAs have consistently been among the most kind, thoughtful, level-headed, and principled people i've ever met. along with rationalists they are the only subcultures i've ever encountered who have a strong norm that they can in theory be persuaded to change their minds on consequential topics by good arguments. you should seriously consider the possibility that, for example, the reason tech billionaire dustin moskovitz funds one of the largest EA organizations is that EAs presented him what he saw as the best argument for what he should do with his billions, which is to find a bunch of thoughtful people to think seriously about how to best spend them to most improve people's lives globally. that is extremely laudable and pretty much the best case!
many people are, i think, confused and perhaps frightened about why this relatively small insular ideological group suddenly appears to have such weirdly large amounts of power, especially power wrt AI. you should seriously consider the possibility that it's in part it's because they spent the last 15 years successfully convincing a bunch of powerful people to listen to them using Facts and Logic, and/or that they themselves decided to acquire power using Facts and Logic (such as arguments that AI would be a big deal and thus one should start an AI company, invest in an AI company, or join an AI company). there is a fairly public history here which it is trivial for you to get an LLM to explain to you which you could learn!
EAs are (mostly, with of course infamous exceptions) trying to do good things for good reasons and are responsive to thoughtful serious critique. they want thoughtful serious critique! they will listen to thoughtful serious critique! you do not have to call them names and schoolyard bully them!
Wild. Researchers suspected AI was dangerous. They tried hard to prove that and failed. They thought it's ok. After Huggingface incident they tried the same setup, and, lo and behold, it replicated!
AI Alignment is HARD.
I was on the communications & policy team at Google DeepMind from 2018 - 2022.
When I first joined GDM, external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization. If asked about existential risk, researchers were PR trained to respond along the lines of: "It's not useful to engage in that kind of alarmism. Some people confuse AI with movies like Terminator -- that's simply not the reality. The AI we develop will be safe by design. After all, we're building it!" And then steer conversation towards beneficial applications in health, climate, etc.
Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.
After months of advocacy, the policy was updated: positively valenced, comms-friendly content on AI safety was permitted (like https://t.co/W631ssnlHV). Note the positive, nice-sounding way that it says "superintelligence could lead to human extinction."
The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it's a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.
We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet.
METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. https://t.co/2f3ypwLPUr
CancerBench: the frontier model cancer cure benchmark.
AI lab CEOs keep talking about curing cancer, so I made a benchmark.
One metric: how many types of cancer has your model cured?
All models are currently tied at zero.
It’s time to hillclimb!
https://t.co/SeEJz0FFYv
If you're thinking of moving into AI safety, there are various excellent non-profit research organizations. They generally pay very well and some try to match AI lab salaries. They have generous compute budgets (and increasing fast).
Here's a quick list of those I'm most familiar with:
@redwood_ai@ApolloResearch@farairesearch
METR
ARC
UK AISI (UK Government, lower pay but very valuable)
Resolution
@CAIS
My organization (https://t.co/Pwkw13jeFD) will also run a hiring round soon.
@FrancescoSacco1 Thank you, wonder if I'm missing on this. Prince has a nice chapter on designing loss functions in "Understanding Deep Learning", which mentions the same technique, I believe.
@melissa I believe. Hans explained how this opening line happened prior, with a different move order. Magnus was uncomfortable in this game, suspicions of Hans clearly affected his quality of play. We have another case where Magnus was uncomfortable, played subpar and lost to Suleymenov.
@S1r1u5_ Completely wrong premise. AlphaGo moves are understandable for professionals. Move 37 originality was blown out of proportion, the idea was pretty clear, humans just had slightly wrong eval. Redmond was surprised more because of move translation issues, less by the move itself.