David Atkinson

PhD student @Northeastern's Bau Lab. Working on AI interpretability. Previously @EpochAIResearch.

[Joshua Achiam @jachiam0]

"Strategic human capacity reserve" - four words that we're going to hear a lot more after the next few years. If government wasn't asleep at the wheel on AI, preparing to build this would be a key national security and economic policy priority.

[Julian Gewirtz @JulianGewirtz]

A notable new line of attack from a Chinese Central Television social media account, entitled "Anthropic Has Contracted the 'American Disease.'" A few points:

1. The "American Disease" is part of a longstanding Chinese critique of American society and American power. Beijing has been talking about the so-called "American Disease" for years. Leading international relations scholar Wu Xinbo even gave a whole lecture on the concept in 2020, where he defined it as "the long-standing and intractable problems and negative phenomena in the political, economic, social, cultural, and foreign relations fields of the United States, which hinder the healthy functioning of the American nation and society, damage America's international image, undermine America's relations with the outside world, and ultimately weaken America's national interests." This new article is constructing a Silicon Valley-specific version of this prior idea.

2. Anthropic is the primary bogeyman, but this critique is far broader. Ahead of the expected U.S.-China AI talks next month, Beijing is joining some in the Trump administration in making Anthropic a bogeyman—although for quite different reasons. But they are also criticizing the entire U.S. AI ecosystem. The commentary calls Anthropic "entirely a monster bred by the American path." And it adds: "The problem is that America’s own frontier models have already developed in a distorted way... the United States is trying to turn the 'security boundaries' it has drawn into the default rules for the entire world. The 'controls' proposed by the United States are, in essence, an effort to have China accept an order whose rules are defined in part by American companies."

3. The commentary is trying to clarify China's position—and set the table to China's advantage—ahead of the potential upcoming talks. It argues that the framing of "security" needs to be resolved in a "reasonable way"—meaning on terms acceptable to China—or else the U.S.-China negotiations will be "meaningless." It holds out the prospect of joint testing and joint restrictions on "capabilities that could genuinely cause serious harm," but that cannot be based on "unilateral declarations by a particular country or company."

4. Beijing has multiple audiences for this commentary. Of course, this is nationalistic fodder for domestic audiences. But, building on Xi's speech in July positioning China as a global alternative to the American AI ecosystem, it's also part of an expansive geopolitical vision. AI competition is not simply about technology, this commentary emphasizes, but about global power and ideological competition. As powerful open weight models like GLM 5.3 are released, Beijing is working intently to frame them as democratizing access to advanced AI, rather than emphasizing the other side of the same openness: making increasingly capable models—including models with advanced cyber capabilities—available for potential misuse worldwide. And a third important audience for this commentary is Washington itself. Based on reports that limited planning has occurred to date for the U.S.-China AI talks, I see this as sending a message squarely to the Trump administration, almost negotiating in public for the upcoming dialogue.

Thanks to @niubi for translating it so quickly.

[Elizabeth Barnes @BethMayBarnes]

One might think "maybe it's ok if models aren't good at doing fuzzy research tasks right now - doing high-quality research and data analysis is a bottleneck to the swarm being able to do do anything that destructive; if subagents give misleading summaries of the work they've done it will limit what AIs can do autonomously. Once models are really dangerous they'll also be much better at these qualitative research tasks." But I'm concerned models might become much more helpful to other AIs vs to humans. Multi-agent training plausibly incentivizes the model to, in a swarm context, do high-quality research and summarize things in ways that other agents can understand (because that’s what leads to collective success on verifiable tasks). Whereas in a human chat/assistance context, the training signal is closer to "produce something that looks superficially good to humans/grader models", not something that causes the human to succeed at a downstream task. We might hope to be able to make human requests look like requests to subagents, but this is maybe hard if the models talk in their own weird dialect and we don’t really know how to translate our requests. I'm pretty excited about directions around "train the model to help the human understand what's going on, such that the human succeeds at a downstream task", although this requires avoiding the failure mode of models just giving the human a list of very specific instructions that they don't understand but that solve the task.

[Bronson Schoen @BronsonSchoen]

We studied o3 / o3-before-safety-training in a ton of depth, it’s what led to:
* https://t.co/eX8L0JDDOS
* https://t.co/z0Ld3L53dr

The best model organisms are just actual training / potentially failed variants / etc, would love to see more research from Ant here as well w.r.t. what’s going on with their models during training. This also sidesteps a bunch of validity concerns.

My impression from system cards / risk reports is that there’s no shortage of “weird thing we saw that we didn’t have much time to investigate in depth” which seem like good candidates for study.

[David Atkinson @diatkinson]

@tylercowen https://t.co/6Vl9k7skFj

[David Atkinson @diatkinson]

If I were a grad student in an intellectual history-friendly department, I'd have Claude neck-deep in the extropians listserv right now

[David Atkinson @diatkinson]

@CharlesD353 @reconfigurthing Possible explanation here is that they’ve stopped doing ~2 OOM compute scale ups for each major version number

[Transluce @TransluceAI]

How could we train an AI model that was very good at overseeing another model: catching reward hacking and sandbagging, predicting unwanted behaviors or fine-tuning effects, etc.?

We propose a new approach for doing this at scale: oversight foundation models.

[Neil Chowdhury @ChowdhuryNeil]

nvidia et al. argue open weights improve safety by letting researchers find vulnerabilities. others argue they enable misuse

but the choice is not binary!

a third option is open access: white-box access for independent researchers, without broadly releasing weights 🧵 (1/)

[Dave Banerjee @DaveRBanerjee]

One underrated fact during this HuggingFace <> OAI debacle: HuggingFace was unable to analyze the logs using frontier models because of the cyber safeguards kept activating and blocking the blue team!!!

This meant the blue team (HuggingFace security team) literally had to downgrade to an open-weight model (GLM 5.2) to do log analysis

Since the cyber safeguards on OAI and Ant models couldn't distinguish between legitimate blue team log analysis versus red team log exploitation, the closed-weight models all REFUSED

I'm not sure what the takeaway is, but if you are a blue team and you don't have special access to model via programs like Project Glasswing, you're in a rough spot...

[David Atkinson @diatkinson]

@DaveRBanerjee Prompted by (2): I wish there was more work on what a granular capabilities access regime should look like.

Ideally, I'd be able to prove to Ant that I am not a bad actor, and that would give me some ability to, e.g., create non-frontier model organisms

[David Atkinson @diatkinson]

I’d be surprised if “knowledge of LW thought” was the relevant difference here—how many people at NeurIPS do you think have heard of inst conv? (Also, to the extent that median ≠ published-in-top-tier, it seems like the second group is a better proxy for experts.)

On timelines, sure, always an issue with these comparisons. I wasn’t reading Nate’s original 5% vs 50% “total risk” claim as being time-restricted though.

[David Atkinson @diatkinson]

@nabla_theta @So8res This seems hard to reconcile with the @AIImpacts survey https://t.co/NoGLD3hz8U, despite matching my intuitive sense of ML researchers

[Jacob Andreas @jacobandreas]

If you're excited about Anthropic's J-space work, definitely worth checking out the original paper on Jacobian lenses by @evanqed and @arnab_api! https://t.co/gttqzgB4VB

[Amiri Hayes @amirihayes_]

What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs.
Swap them in and the model barely notices.

See our experiments here: Explaining Attention with Program Synthesis [https://t.co/tkFopEYtaV]

[Kellin Pelrine @KellinPelrine]

Humanity's ability to know, reason, judge, and act well is the foundation of science, democracy, crisis response, &amp; management of AI itself.

AI poses serious risks to that foundation.

New paper on epistemic risks by 30 experts calls for attention to this. Link in thread.

[Jason Abaluck @Jabaluck]

Models are improving at forecasting (new analysis linked below).

To regulate frontier labs, we should pilot "evaluator" models to review internal code and forecast catastrophic (conditional) risk over time, with mitigations tied to risk once forecasting ability is validated.

[Sheridan Feucht @sheridan_feucht]

Neural networks have beautiful feature geometry, but do they have mechanisms that actually interface with those structures?

At @GoodfireAI this spring, we discovered one: a re-usable addition mechanism that reads/writes to Fourier features from prior work. 🧵

[Lee Sharkey @leedsharkey]

My team at @GoodfireAI has been cooking up a new way to do interpretability: decompose a language model’s weights, not its activations.

Our decomposition natively handles attention (!) and behaves less like a lookup table and more like a generalizing algorithm. (1/6)

[Pierre Beckmann @BeckmannPierre]

New paper with @PatrickButlin, from my time at @MATSprogram . We propose two new candidates for LLM individuation: the (virtual) instance-persona view and the model-persona view. 🧵

[Uzay Macar @uzaymacar]

🧵New Anthropic Fellows research: We studied mechanisms of "introspective awareness" in LLMs.

LLMs can sometimes detect steering vectors injected into their residual stream. But is this worthy of being called introspection, or attributable to some uninteresting confound?👇