Duc Manh Bui

Ph.D. Candidate at University of Technology Sydney (UTS), Australia

[Duc Manh Bui @Buimanh22]

@johnschulman2 100% agree

[John Schulman @johnschulman2]

OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?

[elie @eliebakouch]

we replicated anthropic jspace analysis on @thinkymachines Inkling new 1T model!

it seems to be an outlier: where other models split into near-orthogonal sensory/workspace/motor blocks, inkling keeps roughly one geometry across the whole stack (early-late CKA ~0.8 vs ~0.5 elsewhere)

we also look computed the J-space of @poolsideai's laguna XS 2.1 in bf16 vs nvfp4 to test the impact of quantization. result: almost none. the quantized model has the same jlens space as the non-quantized one

both jspace checkpoint are up on hugging face!

[Tim Li @LiTianleli]

Incredibly proud of the team. After countless late nights, Inkling is out, and I especially want to highlight the post-training stack and RL recipes behind it.

A few of my favorite details:

We scaled our largest RL run to 30M+ rollouts and thousands of continuous training steps—with no collapse, no restarts, and stable KL and entropy throughout. Reasoning performance improved log-linearly from the SFT initialization all the way to the released checkpoint. A number of innovations under the hood made this possible, and the result is a strong testament to our post-training technology.

We trained controllable thinking effort directly through RL. By varying the system message and per-token cost, the model learned to trade off tokens and performance on demand.

We also saw an emergent shift in reasoning style: as RL progressed, the chain of thought became increasingly compressed, shedding grammatical overhead. Inkling reasons like a caveman mathematician—a distinctive style unlike that of other open-source models.

We’re also previewing Inkling-small today and plan to release it very soon. It is exceptionally capable for its size, and we expect the community will find it broadly useful.

Building a simple, stable, and scalable RL stack in such a short time was something few thought possible. This team proved otherwise.

[Duc Manh Bui @Buimanh22]

@karinanguyen Quite a long time since one America's frontier lab release such a good open-weights model :-)

[Duc Manh Bui @Buimanh22]

@johnschulman2 Congrats John and the team :D

[John Schulman @johnschulman2]

Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it.

[Penghui Qi @QPHutu]

This is really a nice work 👍

Using rejection sampling to turn a deterministic clip into a stochastic one, clever and elegant! 💯

[John Schulman @johnschulman2]

We started Thinking Machines a year and a half ago with a couple of instincts: that people should have much more ability to customize models and do research on them, and that even as AI becomes more autonomous, there's a lot more to build to make humans and AIs work well together.

A lot has happened since then, especially the massive progress in agents, so we wanted to revisit those instincts in light of everything we've learned, argue about them, and write down what we actually believe now.

This is where we landed after a lot of debate. I'm happy with it!

[Lilian Weng @lilianweng]

new post on harness engineering for AI self-improvement: https://t.co/ZYvGfVs61k

It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.

Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

[Lilian Weng @lilianweng]

A super long overdue (3+ years?) post on scaling laws.

Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.

The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.

https://t.co/HP26eJvjHB

[Zhihu Frontier @ZhihuFrontier]

Why Would GLM-5.2 Move Away From GRPO?
🌟Insights from Zhihu contributor 九老师

TL;DR: GLM-5.2 dropping GRPO does not mean GRPO is “bad.” It means the assumptions that made GRPO attractive for short LLM RL tasks may no longer hold for long-horizon agentic tasks. When rollouts get longer, environments get noisier, and credit assignment gets harder, PPO + value modeling starts looking useful again.

The key question is not simply “why did GLM-5.2 stop using GRPO?” A better question is: why did GRPO become useful for LLM RL in the first place?
If the reasons that made GRPO attractive no longer hold, then going back to PPO becomes natural.
GRPO can be understood as a sampled-baseline method. Instead of training a separate value model, it samples multiple responses for the same prompt and uses the group average as a baseline.
That is elegant. You get a relative reward signal without paying for a separate critic. In short tasks, this is very appealing.

But there is a tradeoff.⚖️
PPO uses a learned value function, or critic. This critic is expensive and harder to tune. It also has its own problems: the policy keeps changing, so the value model is always trying to follow a moving target. That can introduce bias.
GRPO avoids that by using an up-to-date sampled baseline. It is closer to low-bias, but it tends to have higher variance.
For early LLM RL tasks, that tradeoff made sense:
• Rollouts were short
• Final rewards were clear
• Memory savings mattered a lot
• Multiple samples per prompt were manageable
• Math/code tasks were relatively easy to verify
That is why GRPO worked so well for many short, verifiable reasoning tasks.

But long-horizon agentic tasks change the game. 🎮
A long agent task can look much more like a game environment:
• Many steps
• Tool calls
• Partial progress
• Delayed failure
• Noisy observations
• Intermediate rewards
• Wrong action penalties
• Context compression
• Different paths to the same final answer
This is where GRPO starts to struggle.

The biggest issue is credit assignment. In GRPO, the final reward is applied broadly across the whole trajectory. If a task succeeds, many tokens get rewarded. If it fails, many tokens get punished.
But in a long task, that is too coarse.

Maybe the first half was bad, but the final recovery was good. Maybe one tool call at step 30 caused failure at step 100. Maybe two successful trajectories are not really comparable because one used 4K tokens and another used 200K tokens with heavy tool use and context compression.

GRPO sees the final outcome. It does not naturally know which step actually mattered.
That creates high variance.
In short tasks, group comparison works well. In long tasks, group sampling can collapse into two bad cases:
1. All samples fail
The whole expensive rollout gives almost no useful training signal.
2. Only one sample succeeds
That single success may be luck, but GRPO may treat it as a strong positive signal and over-reward the trajectory.
Both are dangerous for long agentic training.
This is where PPO’s critic becomes valuable again. A value model can learn expected value under noisy states. It can provide denser feedback before the full rollout ends. It is more expensive, but it helps with long-horizon credit assignment.
So the author’s view is: GRPO is not being rejected because it was wrong. It is being outgrown by the task format.

For short, deterministic, verifiable tasks, GRPO remains strong.
For long, noisy, tool-heavy agentic tasks, PPO-style value modeling may simply be the better fit.

The “compaction problem” mentioned around long contexts is likely more of a symptom. The deeper issue is that GRPO’s weaknesses become costly when trajectories are long and states keep changing.
Could GRPO still work? Yes, if paired with a strong Process Reward Model. The author points out that DeepSeek MathV2 uses this direction. Process-level signals can help fix GRPO’s sparse-reward weakness.

But without that, returning to PPO makes sense.
🎯The bigger takeaway:
GRPO saved the value model. PPO brings it back.
GRPO’s main advantage was efficiency. It removed the critic and saved resources. But for long-horizon agentic tasks, the critic’s ability to generalize and assign credit may be worth the cost again.
In the Agent era, RL for LLMs is becoming less like solving a short math problem and more like training an agent to play a long, noisy game.
And for that world, value models may still be the soul of RL.

🔗Full Reading (CN):
https://t.co/hf1GsDBc3e

[Alisa Liu @alisawuffles]

I'm joining OpenAI next week!🥹 The job search turned out to be really challenging but also super rewarding, so I wrote a small blog to share what I learned along the way and hopefully make the process a little less mysterious for the next person. https://t.co/6FigSBdenD

[Penghui Qi @QPHutu]

My take on why PPO is a better choice than GRPO for long-horizon tasks:

1⃣ the group-level synchronization becomes very challenging to the training infra, both sync (long-tail) and async (staleness).

2⃣ group or critic is just a matter of variance reduction, the effect of variance reduction depends on the covariance. For group-based advantage, the covariance between final reward and group average decreases as the sequence goes longer. (the orange curves in the figure below)

image from: https://t.co/x4IOr1edgk

[Duc Manh Bui @Buimanh22]

@mert_gulsun @johnschulman2 Although OPD has been quite hype for months on dense signal but maybe the bound of teacher and lack exploratory comparing to RL may leads to the recall of original actor-critic PPO :-) with more scaling and good RL infra

[Duc Manh Bui @Buimanh22]

@mert_gulsun @johnschulman2 Based on thinky’s blog, the problem is the sparse reward signal for group-based RL. I think that’s why a recall of value net with dense signal on token-level is better, though I remembered OpenAI pointed it out via process reward modeling. Perhaps this time is more scaling

[Duc Manh Bui @Buimanh22]

I’m still a newbie in RL but PPO showed me a way to study this field with better taste. The rejection story behind is surprise but also interesting back to the time before LLM. “A good work will still be widely known and scale to practical regardless of academic submission :-)”

[John Schulman @johnschulman2]

PPO: rejected from NIPS 2017

[Binfeng Xu @billxbf]

“We therefore move from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts, relying on a critic to estimate token-level advantages rather than group-relative comparisons. This single-rollout formulation fits compaction naturally, as it places no constraint on how many traces a prompt produces or on their relative lengths”

It is so true that PPO fits better in multi-trace credit assignment for agenti RL. Still surprised how @Zai_org speeds on carrying this through at scale! 🚀 🐐

[Duc Manh Bui @Buimanh22]

@johnschulman2 Wow quite surprise for that @@ I think the field changed a lot recently but at that time why PPO could be rejected…

[Penghui Qi @QPHutu]

We propose DRPO: a soft version of DPPO🔥

Since PPO, clipping/mask-based trust regions have long outperformed smooth divergence regularization like KL, even though the latter one feels more principled. 👺

We found two missing pieces:👇

1️⃣ Weight the regularizer by |advantage|
- Otherwise, the trust region geometry changes dynamically and optimization becomes unstable.

2️⃣ Use the right divergence
- What matters is not just “regularization”, but the trust-region geometry induced by the gradient. DPPO-style geometry works much better than PPO-style geometry in LLM.

These insights lead to DRPO, which delivers the most robust and best overall performance across algorithms, even outperforming original mask-based DPPO. 🚀

This project is an amazing collaboration with @ExplainMiracles, @NickZhou523786, Wee Sun Lee, Liefeng Bo, @TianyuPang1 . Do follow them if you are interested in this work!

📄 Paper: https://t.co/Un4QNoqUzZ

💻 Code: https://t.co/hPQp91sVUk