Brian Bartoldson

ML researcher

[Brian Bartoldson @bartoldson]

At ICML 2024, we introduced the first scaling laws for adversarial training, improving the *train-time compute* for robustness tradeoff (https://t.co/4Es87BEgDm).

At ICLR 2026, we show how to trade *test-time compute* for robustness efficiently.

https://t.co/4KK0bOU4Hm

[Brian Bartoldson @bartoldson]

Give an LLM a spec: more reasoning ➡️ better spec satisfaction. Even on adversarially attacked data.

But reasoning benefits fade if attacks are stronger (e.g. white-box or multimodal).

Our hypothesis suggests reasoning can stop such attacks.

Toy example in the video. 🧵

[Brian Bartoldson @bartoldson]

Yep, IS (w/ clipping) and larger beta helped stability in some of our experiments. I also have some unpublished stuff suggesting TBA helps deal with FP4 inference off-policyness.

More speculative but potentially also helpful: I like the inference policy as a reference policy because it skips the need for reference resetting, but the best thing to regularize against in my tests is an EMA reference. To first order, regularizing toward the inference policy is regularizing toward an EMA, with regularization strength proportional to staleness (see my website for a short+rough note on this). So dividing each penalty term by its token's staleness gives every token a similar regularization strength, which could help the uneven-staleness issue mentioned in the blog.

[Brian Bartoldson @bartoldson]

@joeyxhong @saurabh_shah2 @ar0cket1 @parth_29 Thank you! TBA is a regression objective too: the following is the loss behind the TBA grad above (16), and with π_ref = μ it's L_CORO, afaict. I think the squared log-ratio penalty is in both, and its gradient is the β(log Λ − mean log Λ) term in 16.

[Siddarth @siddarthv66]

The CORO objective used here is equivalent to the TBA objective proposed in our paper from last year. All of these are variants of PCL and in the on-policy trajectory case yields same gradient as REINFORCE
https://t.co/md7AtjNdiK

[Brian Bartoldson @bartoldson]

@saurabh_shah2 @ar0cket1 @joeyxhong @parth_29 At least some settings (model-data combinations) benefit from IS on top of this kind of regression objective, but in others IS isn't needed.

(IIUC, the TBA objective below matches CORO when you set the reference to be the inference policy and drop IS).

https://t.co/gSImv0TZsq

[Brian Bartoldson @bartoldson]

🔬 Ablation 2: Reference policy resetting

Resetting prevents the stability-adding KL penalty from harming task-reward optimization. This has an effect on performance almost as large as importance sampling's!

[Joshua Saxe @joshua_saxe]

Takeaways from the WSJ article about @HacktronAI using Claude to get into OpenAI's monorepo and issue a pull request (before stopping and claiming their bug bounty)
- how many nation states have already broken in and gone much further and stolen a) algorithmic secrets and b) model weights or c) gotten access to user data; these are fair public interest questions
- how many have implants in / are dwelling in the openai network as I tweet this
- how pervasive is this level of softness to pentesting across all the labs and how far are the labs from the right operating point in the security and r&d friction trade space (probably pretty far it seems)
- given the hacktron folks used anthropic's models to pull this off what's the real public safety ROI of anthropic's cyber guardrails; they add friction for legitimate cyber defenders (like our developers at my startup!) but it appears with a bit of work you can use them in actual breaches as happened here
- as the article says, the wsj folks and @S1r1u5_ had me do neutral technical review of the kill chain here pre publication; impressive from a human angle (@HacktronAI reminds me of the best of my generation of hackers I looked up to as a kid!) but also from what the models can do; between this and the openai/hf thing, I'm emotionally in a place where I feel my world as a security person is being turned upside down in slow motion with respect to what's coming
- elite persistent hacking is becoming rapidly democratized. this is coming like a freight train. we need to harden the world's code and infra as fast as possible and today's non automated methods don't stand a chance of cutting it; the world is a soft target. I continue to be unsettled but very glad I left my comfortable job at Meta to do our automated posture hardening startup

[will brown @willcb]

i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking.

but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance.

ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time.

this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there.

the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.

[Brian Bartoldson @bartoldson]

18 months before the Hugging Face hack:

"On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, cross-lab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat."

Now on arXiv.
https://t.co/t7mc46ndgv

[Brian Bartoldson @bartoldson]

18 months before the Hugging Face hack:

"On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, cross-lab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat."

Now on arXiv.
https://t.co/t7mc46ndgv

[Brian Bartoldson @bartoldson]

@kalomaze oai uses self-play for prompt injection robustness https://t.co/CFayyGgqnL

[Anthropic @AnthropicAI]

New research: Training a Misaligned Reward Seeker

What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.

In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.

Read more: https://t.co/gs2ZjYkPan

[Siddarth @siddarthv66]

LE CRITIQUE: Privileged Value Functions for LLM Reinforcement Learning

Value functions for LLMs have been grossly underutilized. I'm excited to finally share my work on Privileged Value Functions, done @MistralAI !
Our paper includes 2 massive improvements to value functions:

1) Privileged Value Functions: Turns out value functions can use hidden context unavailable to the policy for improved token-level RL signal! Condition them on reference solutions, verifier rubrics, or even other group samples and rewards. No more hacky self-distillation necessary! (being a tad hyperbolic)

2) TETHER: Do not trade off group and value baselines, you can just do both! We introduce a new baseline that adaptively trades off between the group mean and value baselines adaptively based on value function reliability.

🌐 https://t.co/i0HfYbVMUX
📄 https://t.co/rPDnf6oOBQ
Code: https://t.co/y54smOwqPz
🧵below

[Psyho @FakePsyho]

The results on LongCoT are very surprising
- RLM loses to codex
- Opus+CC loses to GLM+pi

I guess (entirely by accident?) they found an Achilles' heel in claude code. Anthropic should be thankful.

[ethantsliu @ethantsliu]

revisiting a paper from early 2025: how should we post-train LLMs efficiently? standard on-policy RL forces data gen and policy updates to happen sequentially, choking hardware utilization and wasting compute time.

instead, the authors decouple exploration from learning by introducing Trajectory Balance with Asynchrony (TBA), using the objective (log z (x) + log pi_theta (y | x) - log reward)^2; z (sampled avg reward/prob from batch) acts as a conversion rate to force the probability to match the reward. this allows a central "trainer" node to continuously learn from a diverse replay buffer that multiple "searcher" nodes populate in parallel

using a mix of reward/recency-prioritized sampling, they calculate pushing this to the max. by aggressively scaling parallel data gen while learning off-policy, this recipe beats baseline accuracy while running 4-50x faster compared to sync methods!

[Sumeet Motwani @sumeetrm]

I'll be presenting 3 papers at ICML 2026🫡

h1 (Spotlight) trains models to reason over longer horizons using curriculum RL over composed short-horizon data. This allows models to generalize to harder tasks and improves performance even at high pass@k.

LongCoT isolates and benchmarks long-horizon CoT capabilities, pushing models to reason over tens and hundreds of thousands of tokens in an output.

Rubric Curriculum RL uses the generation-verification gap in non-verifiable domains to allow models to self-improve on tasks like creative writing and planning. More about this project to come soon!

Very grateful for amazing collaborators @CharlieLondon02 @bartoldson @DanielNichols10 @AmyPrb @hammh0a @jackcai1206 @bkailkhu @casdewitt @sytelus @riashatislam @philiptorr, and especially co-leads Alesia and Tejas!

If you're interested in these topics, my DMs are open :)

[Brian Bartoldson @bartoldson]

@cthorrez @dylanmatt Agree that it's not their raison d'etre, but perhaps a side quest? Showing how things could go wrong in order to inform policymakers was an explicitly stated motivation for https://t.co/jUNRQo8KgJ

[Brian Bartoldson @bartoldson]

The robustness/accuracy you get from scaling is predicted to peak at 90%. However, it seems like 90% robustness/accuracy on this particular task may be equivalent to 100% reliability.

This discrepancy arises because of a weakness in the CIFAR-10 l_inf norm threat model our field has been studying: for a small subset of images, there exist small l_inf norm perturbations that, surprisingly, change the appropriate label.

In particular, we found that 10% of attacked data doesn't fit the definition of a "jailbreak". On this 10% subset, the attacker's change to the input image -- while successfully causing NNs to change their prediction, in line with the attacker's goal -- actually makes the original label for the image a worse description of the post-attack changed image than the new label that NNs assigned to the post-attack changed image (according to a human study we did). I.e., on that 10% subset, the NNs are apparently operating reliably by changing their label, and the attack is not really tricking them. In sum, the evidence suggests that hitting 90% robustness/accuracy could correspond to 100% reliability. Supporting this, we found that humans appear to have a 90% limit on this benchmark too, for the same reason (use of label-changing attacks).

Note that, under a more careful threat model -- where attacker perturbations never change the appropriate label/response -- 100% reliability would imply 100% robustness/accuracy. The field has been using the less careful threat model with the aforementioned issue because it's computationally efficient to compute an l_inf norm bounded perturbation, and much more difficult to compute a label-preserving perturbation (you need a human labeler in the loop for this).

[Micah Goldblum @micahgoldblum]

We trained language models that compress massive contexts into tiny latent representations. Latent Context Language Models (LCLMs) outperform existing KV cache compression methods on the latency/accuracy frontier. 🧵1/10

[FAR.AI @farairesearch]

Recordings and the full recap from the Assurance and Verification of AI Development (AViD) Workshop are live. We co-hosted with @ai_risks on May 17, colocated with IEEE S&P. The question: how do you generate trustworthy evidence about an AI system without unrestricted access to weights or infrastructure? 1/6

[Brian Bartoldson @bartoldson]

@saprmarks Very interesting, thanks for sharing these! It seems like a lot of recent robustness/alignment works are leveraging pretraining interventions -- another one: our ICLR 2026 paper finds that specs can stop white box attacks if the right training set is used https://t.co/4KK0bOU4Hm

[Brian Bartoldson @bartoldson]

Give an LLM a spec: more reasoning ➡️ better spec satisfaction. Even on adversarially attacked data.

But reasoning benefits fade if attacks are stronger (e.g. white-box or multimodal).

Our hypothesis suggests reasoning can stop such attacks.

Toy example in the video. 🧵