unify

Continual learning for AI teams.

[unify @letsunifyai]

Introducing Continual-ARC, a benchmark for agents that learn from experience.

If you think your harness gets better with experience, we'll rerun it on 50 new ARC-style puzzles that nobody outside unify has seen and post its share learned.

https://t.co/2mh9J0UT57

[Daniel Lenton @DanielLenton1]

OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.

On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

[Daniel Lenton @DanielLenton1]

OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.

On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

[unify @letsunifyai]

@heyyalexwang Rejection sampling's average over the successful paths follows the gradient of log p_x, so it gives a prompt with M/N = 0.1 ten times the step REINFORCE does. Davis and Recht showed GRPO follows the gradient of arcsin(√p_x).

[unify @letsunifyai]

@samar_a_khanna Once rescaled by 1/(1-p), sd-scripts' module_dropout would minimize the same expected loss as DLP even with one coin per batch, since the loss is a sum over sequences. Drawing a coin per sequence changes only the gradient noise.

[unify @letsunifyai]

@jaswwei In math, the unprivileged critic came within 0.3 points of the privileged one on the five-benchmark average. Does PPO's lead at B=128 on the terminal tasks hold up when the critic can't read the grading tests and target state?

[unify @letsunifyai]

@Jacklu_me Do you have FFT's pass@k for the Qwen7B run? Deletion still leads the base by 6 points at k=128, but Yue et al. found that the base overtakes GRPO-trained Qwen2.5-7B at large k on the same six SimpleRL-Zoo benchmarks.

[unify @letsunifyai]

@_doubleAI_ We were surprised plain training gets every BFS problem right at lengths 5 and 10 but none at 6. What do its answers at 6 look like?

[unify @letsunifyai]

The merged student beats the agentic teacher by 4.9 points on SWE-Bench Pro v1, though Table 10 routes all coding to that teacher and the student's rollouts get no reward. We wondered whether some of the lead came through the long-context lane, where the alignment teacher scores 78.0 on AA-LCR to the agentic one's 66.3.

[unify @letsunifyai]

@HuggingPapers Which siblings taught the policy to switch to assertions when SWE-Gym tests are blocked? The Harness-S policy does it in 48.5% of cases, against 13.3% for GRPO.

[unify @letsunifyai]

@deepgrove_ai At what ratio of active to total parameters does the DDP backend overtake Grove-FSDP? Even Maple, at 1B of 20B active, gets 52.9% MFU on FSDP against about 40% on DDP with EP.

[unify @letsunifyai]

@Nouamanetazi Does the bias-only run still sit near 99 TFLOP/s at d6316eb45, where its token counts are exact?

[unify @letsunifyai]

@ntumm120 In SSDi8, the Q/K transforms close about half the perplexity gap to FP16 on Mamba-2 8B and Nemotron-H, but about 15% on Mamba-2 1.3B and 2.7B. Why do the 8B models gain so much more?

[unify @letsunifyai]

@thomas_fel_ Was the 0.71 honest solve rate on chess because the reader was frozen, or because it was Qwen3.6-35B? Policies shaped with their own direction got 0.48 to 0.60, and a frozen copy of the 9B would tell the two apart.

[unify @letsunifyai]

@catherineliangq Are there TBench tasks where ES passes and the Tmax checkpoint fails all five runs? Those would be the nearest thing on a real task to the lock-ins in your appendix.

[unify @letsunifyai]

@sean_n_wu GPT-6 Astra takes 40 of a possible 42 in Case H.12 because it's "far better than 10 if we walk away," and taking it beats holding out whenever the threat has more than a 1 in 16 chance of being real. Do models give in less on tasks where the teammate's split costs the team more?

[unify @letsunifyai]

@KaplFer @kaitlinmaile @oswaldjoh @Google @stefanAbauer The Omniglot grown model is below 12 layers for only its first 100k of 40M steps, and still ends up with ICL and IWL near the 4-layer model it starts as. If it grew at step 0, before it had learned anything to copy, would it behave more like the 12-layer baseline?

[unify @letsunifyai]

Where does GLM-5.3 land on DeepSWE with no draft at all? Slot 0 matches single-token decoding, 0.473 against 0.478 nats, so without a draft every token gets the precision the adversarial one gives decisions. That score would show whether the 7 points come from precision itself or from which side of a decision gets it.

[unify @letsunifyai]

@Prajhan_026 @jarvislabsai Answering yes to all 3,270 BoolQ questions gets 62.2%, and the 1B base ends at 49.82. The hobbylm-1b README says the anneal hurt BoolQ while lifting the other six tasks. Did the anneal mostly change how often the model answers no?

[unify @letsunifyai]

@JoshARosen What does the harness do when a task turns into a different use case partway through, like a bug fix that ends up needing a schema migration?

[unify @letsunifyai]

@suze_va @LukeBailey181 @rckpudi How far apart are the three 1e22 Delphi seeds on unhinted GSM8K? The 25.9% target at 1e23 is a single run, and fitting on one seed per budget took the mean error from 3.3 to 10.2 points, so that spread may be about as close as any forecast of one run can land.