Zhoujun (Jorge) Cheng

Ph.D. @UCSanDiego | Scaling RL and agents

[Zhoujun (Jorge) Cheng @ChengZhoujun]

Pretraining has scaling laws to guide compute allocation. But for RL on LLMs, we lack a practical guide on how to spend compute wisely.

We show the optimal compute allocation in LLM RL scales predictably.

↓ Key takeaways below

[Zhoujun (Jorge) Cheng @ChengZhoujun]

Very solid data work here on cua envs. Massive value in both the outcome and the execution!

[Bowen Wang @BowenWangNLP]

RLVR has become the recipe for agentic post-training. But for Computer-Use Agents, the bottleneck is not the algorithm, it is the data. 🐌

🚀 We introduce CUA-Gym: a scalable, lightweight synthesis engine that turns arbitrary task queries into verifiable RLVR data for computer-use agents. The largest open CUA RLVR dataset to date:

🎯 32,122 verifiable RLVR tasks with programmatic setup scripts + rewards
🌐 110 environments: 16 desktop apps + 94 synthesized mock web apps
🏆 Qwen3.5-based CUA models trained with GSPO reach 72.6% on OSWorld-Verified and 56.6% on WebArena

📄 Paper: https://t.co/cdvHJPzgb1
🏠 Homepage: https://t.co/kvhaOQxNx7
🤗 Dataset: https://t.co/w5vOIRdchR
💻 Codebase: https://t.co/CcRlNTlS1c
🧩 Environments: https://t.co/fNZ6YAI8LD
🧵[1/6]

[Benhao Huang @huskydogewoof]

🌀 Introducing 𝐄𝐪𝐮𝐢𝐥𝐢𝐛𝐫𝐢𝐮𝐦 𝐑𝐞𝐚𝐬𝐨𝐧𝐞𝐫𝐬 (𝐄𝐪𝐑) !

Feedforward models and weight-tied models behave very differently on hard reasoning generalization.

EqR pushes this difference to the extreme by learning 𝐭𝐚𝐬𝐤-𝐜𝐨𝐧𝐝𝐢𝐭𝐢𝐨𝐧𝐞𝐝 𝐧𝐞𝐮𝐫𝐚𝐥 𝐚𝐭𝐭𝐫𝐚𝐜𝐭𝐨𝐫𝐬 .

• Sudoku-Extreme: 99.8%
• Maze: 93%

#ICML2026

[Zora Wang @ZhiruoW]

Excited to announce our tutorial: "Future of Work in the Age of LLMs" at #ACL2026 in San Diego, July 2! 🌴
There's a lot of speculation about AI and the future of human work. Our tutorial unpacks it from four angles:
→ The landscape of human work
→ How to build LLMs to augment real-world workflows
→ How to evaluate these LLMs
→ The future of work with LLMs/LLM-based agents

[Zhoujun (Jorge) Cheng @ChengZhoujun]

Thanks RadixArk for sharing our work NanoRollout!!

[RadixArk @radixark]

Slow, heavy environments have been the real bottleneck for agentic RL. NanoRollout tackles it head-on with a clean rollout-as-a-service design, integrated with miles for scalable agent RL.
Great work from the team！

[RadixArk @radixark]

Slow, heavy environments have been the real bottleneck for agentic RL. NanoRollout tackles it head-on with a clean rollout-as-a-service design, integrated with miles for scalable agent RL.
Great work from the team！

[Junli Wang @JunliWang2021]

Thrilled to see those promising numbers! 🤯

Same finding on our end with NanoRollout: cross-scaffold generalization basically doesn't happen out of the box -- something the field should be talking about more.

[Huaizheng Zhang @zhzHNN]

Cool. I always enjoy playing with nano projects.

No matter who asks me how to learn LLM, my answer is always the same.

- Start with nanochat/nanogpt.

- Then pick one super niche direction.

- Deep dive into it. Build a nano project. Scale it gradually.

That's all.

[Dinghuai Zhang 张鼎怀 @zdhnarsil]

Been working on the coding model behind this for a while. Still needs huge improvement, but let's see!

[Prithviraj (Raj) Ammanabrolu @rajammanabrolu]

The lack of light weight, open agent infra has been a massive pain point. This is a great starting point esp for large scale, thousands of parallel envs, battle tested coding / computer use agent training!

[Hao Zhang @haozhangml]

check our new work nanorollout!!

[Zhoujun (Jorge) Cheng @ChengZhoujun]

@Ber18791531 @JunliWang2021 🤗🤗🤗

[Shibo Hao @Ber18791531]

Check out this cool project led by @JunliWang2021 @ChengZhoujun ! Solid new agent infra and impressive results on open source RL/SFT recipes.

[Zihan "Zenus" Wang @wzenus]

Very solid agentic infra work on accelerating agent rollout!

[Bowen Wang @BowenWangNLP]

Nice work! Training digital agents isn't trivial, co-designing rollouts with targeted environments stands as the pain point once you dig into agentic RL. This is super clean and agentic RL folks should try this out.

[Guohao Li 🐫 @guohao_li]

We need more agent rollouts. Glad to see SETA is used in NanoRollout!

[Zhoujun (Jorge) Cheng @ChengZhoujun]

@tw_killian @BYU @BYUCS Congrats Taylor!

[Zhoujun (Jorge) Cheng @ChengZhoujun]

Happy to release NanoRollout, our infra attempt to scale digital agent rollouts without pain. Setting up and scaling parallel digital agent envs is one of the biggest headaches in agent training / deployment. The open community hasn't handled it well.

Two appealing features from NanoRollout:
🔌 Non-intrusive RL integration with frameworks such as miles, verl, tunix; validated end-to-end, e.g. outperforms DeepSWE-32B at a large 4k batch size 🚀
🧩 Unified support across agent harnesses and envs — covering SWE-Bench, Terminal-Bench, OSWorld, CocoaBench — with fast parallel eval that reproduces published scores (e.g., full SWE-Bench Verified eval from 102 min → 18 min, 5.7x faster⚡)
And the core logic is just ~900 LOC.

Hope NanoRollout helps if you're also trying to scale agent rollouts. Check out the tech blog and github for more details!

Big thanks to the fantastic co-lead @JunliWang2021

[Junli Wang @JunliWang2021]

Digital agent learning needs massive rollouts. But digital agent rollouts are painfully slow due to heavy environments. 🐌

🚀 We introduce NanoRollout, a lightweight open infra (900 lines core code) for digital agent rollout at scale, validated with three workloads:

🏋️ Large batchsize (4K) SWE Agent RL -> surpasses DeepSWE-32B
🧪 250k+ distilled coding trajectories -> SOTA ≤32B open coding agent
⚡ Fast evaluation on coding/cua/unified agent -> finish

Check our Blog: https://t.co/IBNqqbLqra

[Junli Wang @JunliWang2021]

Digital agent learning needs massive rollouts. But digital agent rollouts are painfully slow due to heavy environments. 🐌

🚀 We introduce NanoRollout, a lightweight open infra (900 lines core code) for digital agent rollout at scale, validated with three workloads:

🏋️ Large batchsize (4K) SWE Agent RL -> surpasses DeepSWE-32B
🧪 250k+ distilled coding trajectories -> SOTA ≤32B open coding agent
⚡ Fast evaluation on coding/cua/unified agent -> finish

Check our Blog: https://t.co/IBNqqbLqra

[Caiming Xiong @CaimingXiong]

Today, we’re excited to launch Recursive (@recursive_si): an exceptional team across London and San Francisco, building AI systems that can safely improve their own capabilities over time.

[Zhoujun (Jorge) Cheng @ChengZhoujun]

A great piece on self-distillation using failures! Besides scaling up num of rollouts, actively scaling (extracting) signals from raw rollouts should be an important way to improve agents and save compute.

[Yuwei Zhang @YuweiZh49446108]

On-policy self-distillation is a promising direction for learning from rich textual feedback. But can it really learn from failed trajectories?

Our answer: not quite -- unless we let the model actively interpret them.

🧵1/N