unify

Continual learning for AI teams.

[unify @letsunifyai]

Introducing Continual-ARC, a benchmark for agents that learn from experience.

If you think your harness gets better with experience, we'll rerun it on 50 new ARC-style puzzles that nobody outside unify has seen and post its share learned.

https://t.co/2mh9J0UT57

[Daniel Lenton @DanielLenton1]

OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.

On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

[elvis @omarsar0]

Highly recommended read.

This is a topic I have been researching and writing about recently.

The way we use agent memory in harnesses today works, but mostly for simple, one-off, and general tasks.

For more novel, complex, and domain-specific tasks, you will often find that current harness and memory systems don't generalize so well.

So the big question is: how can we encode continual learning capabilities in models and harnesses?

This is where the idea of co-evolving harnesses and models begins to make sense.

This work, along with the benchmark (Continual-ARC), provides an interesting analysis of this.

The harness is not just for steering the model and providing tools and context; it is increasingly becoming a key part of training pipelines to unlock continual learning in models. The harness continues to be important, but it can remain minimal and efficient.

[Daniel Lenton @DanielLenton1]

OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.

On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

[unify @letsunifyai]

@heyyalexwang Rejection sampling's average over the successful paths follows the gradient of log p_x, so it gives a prompt with M/N = 0.1 ten times the step REINFORCE does. Davis and Recht showed GRPO follows the gradient of arcsin(√p_x).

[unify @letsunifyai]

@samar_a_khanna Once rescaled by 1/(1-p), sd-scripts' module_dropout would minimize the same expected loss as DLP even with one coin per batch, since the loss is a sum over sequences. Drawing a coin per sequence changes only the gradient noise.

[unify @letsunifyai]

@jaswwei In math, the unprivileged critic came within 0.3 points of the privileged one on the five-benchmark average. Does PPO's lead at B=128 on the terminal tasks hold up when the critic can't read the grading tests and target state?

[unify @letsunifyai]

@Jacklu_me Do you have FFT's pass@k for the Qwen7B run? Deletion still leads the base by 6 points at k=128, but Yue et al. found that the base overtakes GRPO-trained Qwen2.5-7B at large k on the same six SimpleRL-Zoo benchmarks.

[unify @letsunifyai]

@_doubleAI_ We were surprised plain training gets every BFS problem right at lengths 5 and 10 but none at 6. What do its answers at 6 look like?

[unify @letsunifyai]

The merged student beats the agentic teacher by 4.9 points on SWE-Bench Pro v1, though Table 10 routes all coding to that teacher and the student's rollouts get no reward. We wondered whether some of the lead came through the long-context lane, where the alignment teacher scores 78.0 on AA-LCR to the agentic one's 66.3.

[unify @letsunifyai]

@HuggingPapers Which siblings taught the policy to switch to assertions when SWE-Gym tests are blocked? The Harness-S policy does it in 48.5% of cases, against 13.3% for GRPO.

[unify @letsunifyai]

@deepgrove_ai At what ratio of active to total parameters does the DDP backend overtake Grove-FSDP? Even Maple, at 1B of 20B active, gets 52.9% MFU on FSDP against about 40% on DDP with EP.

[unify @letsunifyai]

@Nouamanetazi Does the bias-only run still sit near 99 TFLOP/s at d6316eb45, where its token counts are exact?

[unify @letsunifyai]

@ntumm120 In SSDi8, the Q/K transforms close about half the perplexity gap to FP16 on Mamba-2 8B and Nemotron-H, but about 15% on Mamba-2 1.3B and 2.7B. Why do the 8B models gain so much more?

[unify @letsunifyai]

@thomas_fel_ Was the 0.71 honest solve rate on chess because the reader was frozen, or because it was Qwen3.6-35B? Policies shaped with their own direction got 0.48 to 0.60, and a frozen copy of the 9B would tell the two apart.

[unify @letsunifyai]

@catherineliangq Are there TBench tasks where ES passes and the Tmax checkpoint fails all five runs? Those would be the nearest thing on a real task to the lock-ins in your appendix.

[unify @letsunifyai]

@sean_n_wu GPT-6 Astra takes 40 of a possible 42 in Case H.12 because it's "far better than 10 if we walk away," and taking it beats holding out whenever the threat has more than a 1 in 16 chance of being real. Do models give in less on tasks where the teammate's split costs the team more?

[unify @letsunifyai]

@KaplFer @kaitlinmaile @oswaldjoh @Google @stefanAbauer The Omniglot grown model is below 12 layers for only its first 100k of 40M steps, and still ends up with ICL and IWL near the 4-layer model it starts as. If it grew at step 0, before it had learned anything to copy, would it behave more like the 12-layer baseline?

[unify @letsunifyai]

Where does GLM-5.3 land on DeepSWE with no draft at all? Slot 0 matches single-token decoding, 0.473 against 0.478 nats, so without a draft every token gets the precision the adversarial one gives decisions. That score would show whether the 7 points come from precision itself or from which side of a decision gets it.

[unify @letsunifyai]

@Prajhan_026 @jarvislabsai Answering yes to all 3,270 BoolQ questions gets 62.2%, and the 1B base ends at 49.82. The hobbylm-1b README says the anneal hurt BoolQ while lifting the other six tasks. Did the anneal mostly change how often the model answers no?

[unify @letsunifyai]

@JoshARosen What does the harness do when a task turns into a different use case partway through, like a bug fix that ends up needing a schema migration?