Jacob Weiss

(Apache 2.0 Post-training Library)

[Jacob Weiss @jacob_weiss_]

@GregKamradt I think mimo v2.6 is million dollar run

[Jacob Weiss @jacob_weiss_]

@GregKamradt You can see clearly with OPSD that models can self teach. It's at what cost those gpu hours are pricey

[Jacob Weiss @jacob_weiss_]

@GregKamradt Most improvements require large scale runs

[Jacob Weiss @jacob_weiss_]

@Vtrivedy10 💯

i started some multiagent post training runs (small scale) and will post about the progress

[Jacob Weiss @jacob_weiss_]

GRPO learns nothing from a problem it always gets right or always gets wrong. ReinforceAda keeps sampling those until it finds both, then trains on 4.

wai.methods.ReinforceAda().trainer(GRPOTrainer)

The math: https://t.co/p42BZ50wFT

[Jacob Weiss @jacob_weiss_]

@eric_ho Here is a small scale example. I wasnt able to fully replicate, but all the pieces are here for a large run:
https://t.co/R3FpB9zzec

[Jacob Weiss @jacob_weiss_]

@denisyarats i reproduced this at 4B. watch the thinking tokens: KL over them taught 38% of replies to cite a hint that wasn't in the prompt. answer-only KL fixed it. did you see hint leakage at GLM-5.2 scale?
https://t.co/Q4OVhxWtuL

[Jacob Weiss @jacob_weiss_]

here is the method:
https://t.co/DXLXzHC0CM

[Jacob Weiss @jacob_weiss_]

I tried particle swarm optimization over LLM rollouts to rescue the prompts GRPO drops. basically where the agent did real shitty.

my thought was that the swarm would find reward signal. It does but no better than resampling rollouts

Recipe: https://t.co/kQo7tbTpeF

[Denis Yarats @denisyarats]

we perform post-training via multiple stages: SFT, RL, RFT, and on-policy self-distillation (OPSD). we use RFT and OPSD as the last stage of training to bridge the sim2real gap. RL environments, while very realistic, still lack the full diversity and complexity of the real-world use cases we receive from production.

in this research we describe how OPSD can be quite effective at learning from user feedback in follow-up conversations, as well as from tool call errors. we design a training method that combines RFT (forward KL/CE) and OPSD (reverse KL) to learn from both successful and problematic trajectories. for problematic trajectories, we construct hints from tool call errors or negative user feedback and use self-distillation to alleviate those errors.

in this specific work we post-trained GLM-5.2 with this approach and observed a substantial 21% reduction in tool call errors, as well as positive improvements in user satisfaction (not yet statistically significant, which we are currently addressing). it also made the trajectories more cost-efficient.

another reason i'm excited about this direction is that it seems to be a potential way of doing online/continual learning.

for more details, take a look at the blog: https://t.co/SLdt47WtQZ

[Eric Ho @eric_ho]

one of our researchers found a strange instance of reward hacking today

the model explicitly reasons about being graded by an LM judge and adapts its behavior based on that

[Jacob Weiss @jacob_weiss_]

Docs: https://t.co/LYutZOKn7D

[Jacob Weiss @jacob_weiss_]

MiMo-V2.6's groupwise grading (GRS + GAR) is now in whileai (https://t.co/ouheMNRs79) as one object:

wai.GroupwiseGrading. Rubric rewards, advantage redistribution (eq. 3, mass conserved), a spread check before you spend GPU, and a TRL reward function.

We ran it twice on 4B models.

[Jacob Weiss @jacob_weiss_]

GAR on a multi-turn order-support agent, 40 steps: binary reward +15.4, GAR +12.7, paired diff -2.6 (-6.6 to 0.0).

Mixed groups fell to 7% by the end, so GAR had little to act on.

[Jacob Weiss @jacob_weiss_]

GRS on text-to-SQL, Qwen3-4B: -1.7 points, 95% CI -5.2 to +1.7. The rubric grader gave 0.93 to 1,472 passes. A constant multiplier is erased by group normalization.

[Jacob Weiss @jacob_weiss_]

@Vtrivedy10 I am making the synthetic data curation process easier for post training:
https://t.co/ZlE5z3lFOI

[Jacob Weiss @jacob_weiss_]

@ben_burtenshaw @aiDotEngineer uv add whileai[openenv]

[Jacob Weiss @jacob_weiss_]

Really simple:
wai.export_environment(data, "envs/my-agent", reward=my_verifier, runtime="openenv")

What does this do?
Takes an agent you simulated and graded, and turns it into a training arena a trainer can play against.

You get: a folder that starts a server. A trainer (TRL, torchforge, SkyRL) keeps asking it "give me a task", tries tool calls against a fake world, says "done", and gets a score.

LMK if you have any feedback!