Andrew Wang

PhD student at @jhuclsp

[Andrew Wang @andrewwnlp]

New📄: Skill learning helps agents adapt to new domains. What about making agents more cost-efficient as well?

We introduce SpeedRunner: the first skill learning paper to make cost a primary optimization target 🧵

[Alvin Zhang @ZAlvin39105]

1/8

How do we train a model to adapt to tasks it has never seen, without updating its weights at test time?

With Harness Learning, we train a model to revise harnesses using execution feedback. At test time, the model applies this learned skill to improve harnesses on unseen tasks, while model weights stay fixed.

Improving agent harnesses takes experimentation. We want a model to learn from those revisions and make better edits on unseen tasks.

Models trained with harness learning nearly double their performance on challenging unseen reasoning and multihop question-answering tasks.

[Nikhil Sharma ༗ @nikhilsksharma]

We can't build cumulative knowledge about #AI's impact on humans if we can't reproduce studies and evals

‼️ 97% of recent conference papers had replication issues👇

https://t.co/ZzIt2CiGRj is built to solve this problem

📄 https://t.co/CThkBtU24g

#HCI #NLProc #OpenScience

[N8 Programs @N8Programs]

Announcing our latest research at @jhuclsp: "Convergent Emergence of In-Context Learning Across Modalities"

We show models trained on language, genome, proteins, images, timeseries, and integer seqs all exhibit few-shot ICL - and that their performance is highly correlated! 🧵👇

[Brian Zheyuan Zhang @zheyuanzhang99]

Existing continual learning methods still suffer from catastrophic forgetting. Our recent work found a simple recipe to substantially improve 𝗹𝗼𝗻𝗴-𝗵𝗼𝗿𝗶𝘇𝗼𝗻 𝗺𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻:

Composing multiple continual learning mechanisms extends memory lifetime and raises average final retention from 1.2% to 34.9% across 3 datasets: a 28-fold improvement over naive sequential SFT.

Project page: https://t.co/LReleAxEUh
Paper: https://t.co/jLZWGcUFIK
Code + datasets: https://t.co/uwlTrYKbJ2

[Daniel Khashabi 🕊️ @DanielKhashabi]

The setup here is long-horizon memorization: learning (many) tasks sequentially while retaining knowledge from earlier ones.

The finding is simple: composing existing continual learning mechanisms is a very strong yet simple receipe.

[Daniel Khashabi 🕊️ @DanielKhashabi]

I am probably late to the party. Here I am sharing a few hot-takes on the OpenAI/HF incident as an outsider.

1. Was the emergence of a 'swarm' necessary for the cybersecurity incident?

The majority of the online commentary/discourse is focused around the curious emergence of multi-agent communication (the exploits that enabled emergence of message boards, indirect agent-agent messages, leading to the 'swarm').

There is an implicit conclusion that such society formation was a key driver of what made the attack possible. I disagree. Every exploit in that chain was found by some single agent. It is the misalignment of individual agents that caused this incident. In an alternative world, if OpenAI ran the same experiment with one agent at a time, my bet is that we would have seen similar outcomes.

A key element of the study, IMO, is the mere existence of a scratchpad (message board), which enabled persistence of findings over time. This would allow any agent (even if it was just one agent) to write notes and read later. So even if the agent gets shutdown/reset, it can pick up the lessons left by its previous efforts.

One may rebut this by arguing that, as the report notes, agents specialized into roles that were useful only to the 'swarm'. Though, a single agent with a scratchpad (message board) can do the same serially.

So did collective effort of the 'swarm' make the attack easier? Even this is not obvious. It's probably more convincing to say that the swarm sped up the attack in wall-clock time (by having many parallel efforts, rather than one sequential thread, hence distributing the load of thinking). But this is a shaky argument if you account for the total compute budget. Again, my bet is that if you allocate the total compute of the swarm to one of their agents, it will be nearly as lethal as the swarm. (One caveat that, a single run has a context limit; but that's exactly why the scratchpad/message board helps).
So my bet is that the swarm bought wall-clock speed, not capability. And probably at worse compute efficiency, given how much of the coordination was agents overriding each other, refusing on ethical grounds, or ignoring each other's pause requests.

2. Was the setup hard to monitor because of the 'swarm'?

Quoting Ryan Greenblatt: "... we don't have good approaches for understanding activity in AI 'swarms'" when discussing this incident, there is an implicit sense that the mere emergence of a 'swarm' makes it harder to monitor the communication. I also find this hard to wrap my head around.

To my knowledge, what makes the problem complex is the sheer scale of reasoning traces. There are simply too many rollouts, and each of them is very long, involving various read/write tool calls. It's just hard to put the pieces together. That's not necessarily a property of a 'swarm'!

Suppose instead of a 'swarm', you have one agent only that has access to the same amount of compute as the swarm. You're going to get trace data that is similarly large and hard to analyze/monitor.

It's possible that the mere existence of 'swarm' is adding more complexity to the data (after accounting for a fixed total compute), though I don't know if we have any quantitive evidence for that.

Overall, it's unclear if 'swarm' is what made the monitorability of this problem *more* challenging. (Happy to be convinced, if anyone has evidence to the contrary.)

Last but not least: big kudos to OpenAI/METR/Redwood for their detailed reporting on this incident.

[William Jurayj @ COLM 2026 @williamjurayj]

We share a fresh perspective on skill learning: beyond just generalization, library learning can help agents handle unseen scenarios at lower costs!

Check out @andrewwnlp’s thread & the paper for more!

[Daniel Khashabi 🕊️ @DanielKhashabi]

Codify recurring behavior into programs, and the agent stops paying an LLM to re-reason every step => cost reduction.

Nice result from @andrewwnlp!

[Andrew Wang @andrewwnlp]

Thanks to my collaborators Zixi Huang, Xiheng Wang, @williamjurayj, @bernaaaljg, @DanielKhashabi, and Nick Andrews at @jhuclsp!

[Andrew Wang @andrewwnlp]

New📄: Skill learning helps agents adapt to new domains. What about making agents more cost-efficient as well?

We introduce SpeedRunner: the first skill learning paper to make cost a primary optimization target 🧵

[Andrew Wang @andrewwnlp]

Overall, we find that symbolic/programmatic skills can make learning cheaper without compromising performance.

See our paper for more details:

arxiv: https://t.co/JduQ7GTrX0
website: https://t.co/jYgUHVml92
code: https://t.co/XUNynI9dbW