Joy Wongkamjan

PhD student @ UMD, Google Intern. RL / Human-AI coop nerds

[Joy Wongkamjan @joywwong]

Our paper CTRL-D is accepted to ACL Findings and will be presented at ACL 2025!

🗓️Poster session: 18:00–19:30 (Level 0 Exhibit Halls X4/X5)

I’m sad I can’t be there, but Jordan (@boydgraber) will! You’ll enjoy learning about CTRL-D from him.

Now… what is CTRL-D? 🔍

[DAIR.AI @dair_ai]

Brilliant new paper from the Qwen team.

It provides insights into where agent training environments actually come from.

Terminal agent trajectories have accumulated at scale while realistic executable environments are scarce.

Environments are what post-training needs, since each one can be re-queried into many verifiable tasks and returns execution feedback, while a trajectory is a single frozen demonstration.

Terminal-Universe reconstructs the environment from the trajectory instead of generating one from scratch.

The tool-execution history in an existing trajectory already exposes the structure and contents of the environment it ran in. Replaying the recorded file operations restores each file to its pre-modification state, giving a partial workspace, and a completion agent then supplies the missing files and dependencies.

They scale the recovered workspaces two ways.

For breadth, mined dependency relations between related environments produce cross-workspace queries spanning multiple codebases. For depth, a single-turn query becomes a multi-round session where a user agent supplies iterative feedback and requirement refinement.

Applied to public terminal agent trajectories it yields 37.3k task-sufficient environments. Supervised fine-tuning of Qwen3.5-27B on that corpus improves Terminal-Bench 2.1 by 11.9 points and EvoCode-Bench v2 MT@4 by 13.8 points.

Paper: https://t.co/GFFVRqfmjZ

[Xudong Han @Xudong07452910]

Netflix 每周要生成几十万条「为什么推荐这部剧」的解释，人类根本审不过来。

于是他们把 LLM-as-a-Judge 真正放进了生产系统。

这篇论文记录了 Netflix 如何搭建一套 Judge 的完整生命周期：建立人工标准、训练 Judge、上线拦截，再持续监控和更新。

其中一个设计叫 RART。

Judge 除了要判断一条解释能不能通过，还要和人工评审在「为什么失败」这件事上对齐。因为上线后，Judge 给出的失败理由会直接交给生成模型，让它重新修改答案。

理由判断错了，下一轮修改也会被带偏。

上线后，每条解释都会经过「生成 → Judge → 修改」的循环，最多重试 3 次；同时每周再抽取约 300 条结果交给人工复查，用来监测 Judge 有没有随着内容变化逐渐漂移。

最终他们做了 5 周、覆盖数千万 Netflix 用户的 A/B 测试。

加入这套 Judge 之后，用户观看未看过内容的比例提升 0.2%，成功从浏览进入播放的 session 提升 0.3%。

数字看起来不大，但在 Netflix 这个规模下已经很有意义。

现在我开始愿意把 LLM-as-a-Judge 看成一种长期运行的基础设施。

上线时对齐一次远远不够。数据会变、用户会变、生成模型也会变，Judge 自己也需要持续被检查和校准。

未来 Agent 系统里的 evaluator，可能也会逐渐拥有自己的「生命周期」。

📎 arxiv: https://t.co/ECvJ9UrDLH

[Tom McCoy @RTomMcCoy]

🤖🧠NEW PAPER🧠🤖
(The result of an 8-year project!)

LLMs seem very different from symbolic systems. Yet LLMs excel in symbolic domains (e.g., language/code/math). How do they do it?

Our finding: LLM representations have implicit symbolic structure!

Link in thread ⬇️
1/n

[Ge Zhang @GeZhang86038849]

Modeling the concept instead of the token is the future.

https://t.co/JtKkddkzdG
https://t.co/ka55uXic2U

[Gill @gurtej__gill_]

Everyone working on RL for LLM agents knows the pain of credit assignment.

A 25 step run fails at the very end, and methods like GRPO blindly punish every single step along the way.

I’ve been reading this new paper and i feel their solution is amazing.

Instead of training finicky critics or burning compute on branch rollouts to guess what went wrong, they just look inside the task’s own verifier script.

The verifier already checks concrete state changes and constraints, so they trace those logical checks directly back to the specific actions that triggered them.

And then shift the training advantage along those exact proof edges.

No extra critic, no sampling overhead, and zero need for the verifier at inference.

And also it abstains whenever the causal evidence is ambiguous rather than simply just guessing, which prevents injecting toxic pseudo-labels into the advantage tensor.

Their ablations on ALFWorld & WebShop show that simply throwing dense atom at the problem doesn't cut it without the actual dependency edges.

I think it’s a super clean and super great idea.

Definitely worth a close read if you’re tackling long-horizon agent training: https://t.co/BmBVPPKXDj

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Really awesome MOPD reading list. Here is also another well-written blog that synthesizes important work in this space (mostly from OSS model reports) for those who are interested: https://t.co/kNc9heqE3T

[Gill @gurtej__gill_]

I spent my morning reading this new paper on batch scaling in LLM Reinforcement Learning.

And i feel it addresses the bad habit of cranking up batch sizes just because our GPUs look underutilized.

The authors simply just separate the systems side from the algorithmic side.

We all know autoregressive rollout generation is heavily memory-bandwidth-bound at low concurrency, so jumping to a larger batch gives you an immediate, satisfying throughput jump (they hit up to 2.29× on fixed hardware).

But because each step now burns far more samples, you easily end up trading away learning efficiency.

Their rule of thumb is that a bigger batch only cuts down real wall-clock time if your throughput gains actually outrun that sample penalty.

The most practical takeaway for anyone running GRPO or PPO is their square root learning rate scaling under Adam.

Across a bounded window (B to 4B), retuning the learning rate kept sample efficiency steady and slashed time-to-target by 29%.

If you leave the learning rate alone, the higher throughput is an illusion and you actually train slower.

Push it out to something like 16B, and the entire advantage falls off a cliff.

It’s a super grounded and disciplined study.

If you’re currently tuning LLM RL pipelines, do yourself a favor and check this out: https://t.co/C2ub89kd24

[Peixuan Han @peixuanhakhan]

🎉🎉🎉Thrilled to release a new paper on LLM RL (collaborating with Amazon AWS):
"Cliff: Learning Process Rewards from the First Mistake"

If you're interested in reward shaping and distilling in RLVR, take a look at our work! Paper: https://t.co/YKsFkqWshT

(More below⬇️)

[Michal Korniak @MichalAKorniak]

Self-supervised RL usually models one action at a time. We extended contrastive RL over action chunks instead, and found large gains: +31.7% offline, +93.1% online. Many explanations exist for why chunking helps. We find a new one: it improves the critic's representations.

[fly51fly @fly51fly]

[LG] How Do Language Models Choose Between Context and Memory?
B Shih, J Winnicki, A Cao [Stanford University] (2026)
https://t.co/bIPNys2AGC

[Shrinivas Ramasubramanian @stablegradients]

Is RL optimizing the right objective? 🤔

Should we maximize mean reward? Best-of-k? Which k?

Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲

We introduce Tail-Likelihood Reinforcement Learning (TailRL).

It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.

🧵 1/n

[elvis @omarsar0]

Banger paper from Google DeepMind and colleagues.

(bookmark it)

A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it.

In other words, if you ask about one detail from a 1M-token conversation the global attention layers re-read all of it, per token.

The usual fix is to guess the relevant tokens first with cheap proxy scores, which still costs O(N) every step. Declarative Attention asks the model instead.

The model declares where it needs to look, inside its own chain-of-thought.

In this way, generation splits into three modes: global reads the full context, focus reads one specific region, and local reads only recent output.

The inference engine parses those declarations the same way it parses tool calls and skips most of the cache read.

On zero-shot on off-the-shelf weights across 15 long-context tasks, attended tokens during decoding drop 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B.

Paper: https://t.co/caC2iKjXGD

Chat with Paper: https://t.co/6TIllXY5nQ

[Andrew Koh @andrewjkoh]

We develop a mechanism design framework for AI alignment and control: https://t.co/7eI8H52s8h

It’s largely conceptual but we offer stylized applications to failure modes (sandbagging, alignment faking), safety practice (scalable oversight, peer prediction), and a way to think about the value of alignment, interpretability, capability, and control.

[Alec Helbling @alec_helbling]

Introducing Flow Reasoning Models.

We developed a recurrent flow-based architecture to efficiently solve structured reasoning problems (e.g., Sudoku).

FRMs apply continuous flows to discrete data and recurrently refine their past mistakes through self-conditioning.

[Shannon Shen @shannonzshen]

We share our early research on building Software World - a "GitHub" run by agents. We deploy agents for Python packages in a dependency chain, and each agent is tasked with collaborating with others and optimizing the package it owns.

[elvis @omarsar0]

Massive paper from Meta.

I like this one because it shows the use of agent harnesses for production-grade recommender systems.

Details below:

This is one of the more convincing agent deployments I've seen.

It runs against a live production recommender serving billions of people and reports A/B results.

Sustaining a recommender is continual optimization work. Content shifts, user behavior shifts, upstream models shift, and the choices governing retrieval, ranking and serving have to be revisited.

Human engineers test those changes through online experiments, which is slow enough that parts of the system go unrevised.

In CORAL, each cycle the agent observes operating signals, reasons over a memory of past decisions and their measured outcomes, and invokes tools including a numerical optimizer that keeps every change inside a fixed operating budget.

The policy improves in context from its own prior actions, with no parameter updates.

Across two large social platforms, the same harness improves engagement at no additional serving cost on one and reduces serving cost without degrading engagement on the other.

Performance improves as the loop iterates.

The guardrail design carries as much weight as the agent. A bounded change budget makes this safe to run against production.

Paper: https://t.co/G46EgVuPMR

Chat with Paper: https://t.co/KlYFT8dAFD

[Ai2 @allen_ai]

Do LLM safety & capability evals measure what they claim to?

We built BenchMIRT to audit them + see which model abilities their Qs actually test.

On BBQ, a social-bias eval, it found the Qs distinguished models more by reasoning ability than safety. 🧵

https://t.co/BQ8WPTu5nR

[John Hewitt @johnhewtt]

As agents perform increasingly complex tasks for researchers, it's important to develop norms for communicating with other humans. Here's my lab's policy on AI in writing and communication, focusing on helping us think deeply and communicate precisely.

https://t.co/vdvM7Ky3Os

[World Labs @theworldlabs]

Introducing Atlas:

The world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D.

Model the world, move the camera, and simulate space & time.

[Fei-Fei Li @drfeifei]

I'm so excited that our @theworldlabs team has achieved a major milestone today! Introducing Atlas - a first of its kind multimodal world model trained from scratch! 🚀

Atlas is capable of generating frames with pixel-perfect camera control, reconstructing large scenes from as few as one single input image, simulating space-time by reframing videos, natively outputting 3D spaces from one or more input images, composing multiple posed images into a consistent 3d world, and more! This is the best camera conditioned world model ever, opening doors to many possible use cases from VFX to robotics. I'm so so so proud of our team!♥️