How much do LLMs’ watermarks distort your output? 🔍
We present SimpleMark: a fast multi-bit text watermark that bounds the distortion of every token to a chosen f-divergence budget. Our watermark’s text distribution stays statistically close to what the model would have written without a watermark while remaining competitive in all benchmarked performance categories.
Harnesses used to be a crutch for weak LLMs. In 2026 and onward, with stronger models, harnesses now offer a mechanism for composing an LLM’s existing and new capabilities to generalize well beyond their training environments.
In our new paper, LM Harnesses are Compositional Generalizers, we explore how various harness abstractions enable LM systems to generalize over task length, task strategies, recursive task depth, and more through composing existing LLM capabilities. These abstractions provide a means for more data-efficient training, especially over long tasks, by keeping trajectories locally in-distribution (LID) — i.e., shaping all individual LLM calls to be in-distribution.
For tasks that take a very long time to roll out, this implies that we can even perform surrogate harness training, meaning we train an individual LLM call instead of doing the very long harness rollout, using purely synthetic tasks, with fake tool calls + sub-agents!
A common idea floating around is that LLMs will absorb harnesses and make them obsolete. We disagree, and argue that we haven’t sufficiently explored what harnesses can provide.
The full paper, logs, codebase, and data are available: https://t.co/AXBjdoqmiz
w/ @SOURADIPCHAKR18@lateinteraction
@a1zhang@lateinteraction@jayden_teoh_ The edited model is now on the Hub: https://t.co/cr0kREszym
Huge thanks to @akhaliq, who rebuilt it from our published logs, re-ran MATH-500 (51.0%).
They also seems to have a live demo: https://t.co/1hzCFubGHw
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: https://t.co/bOWCcFhCVq
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: https://t.co/bOWCcFhCVq
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: https://t.co/K69tPSXe2k
💻 Code: https://t.co/K1G4CAPi7d
🤗 HuggingFace: https://t.co/IQ86GSJn55
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: https://t.co/K69tPSXe2k
💻 Code: https://t.co/K1G4CAPi7d
🤗 HuggingFace: https://t.co/IQ86GSJn55
Hot take: Most autoresearch today is just glorified overfitting.
Current autoresearch pipelines are essentially outer-loop searches for recipes that work in ONE specific setting. Try transferring a modded-nanogpt recipe to a different language or dataset. How much of that "optimal" recipe still holds up?
We need to move beyond static recipes toward inner-loop autoresearch: optimizers, architectures, hyperparameters, etc., continuously adapting DURING training based on the training trajectory itself.
Check out OPAL that enables this! Cool work with @SimulatedAnneal, Jia-Hao Liang, and @vincesitzmann.
Surprising new results:
Giving tools to LLMs makes them less able to say NO!
We tested 11 multimodal LLMs (Claude, GPT, Gemini, ...). With tools, every one fails to refuse harmful requests more often, up to 68.7% more!
Details and possible reasons 🧵
To appear at NeurIPS’26.
How much do LLMs’ watermarks distort your output? 🔍
We present SimpleMark: a fast multi-bit text watermark that bounds the distortion of every token to a chosen f-divergence budget. Our watermark’s text distribution stays statistically close to what the model would have written without a watermark while remaining competitive in all benchmarked performance categories.
5 / This is a joint work with my amazing collaborators at Amazon: Wanrong Zhang, Weitong Ruan, and my wonderful MS advisors at UIUC: @lrvarshney and @alabid_res.
Paper: https://t.co/im5dA1YNW4
Blog post with an interactive demo: https://t.co/5a74ACIdM3
4/ Our experiments show that our per-token distinguishability of tokens containing watermarks adheres to the ε budget which outperforms the previous state of the art, while reasoning performance, perplexity, and message recovery remain competitive.