Yijia Fan

Working on Embodied/LLM/Agents
Intern @Microsoft
Incoming PHD @ MMLab CUHK

[Yijia Fan @YijiaFanCSAI]

Every strong code-as-policy robot agent today runs on a closed frontier model. You can’t fine-tune it, you can’t put it on the robot, and it costs $8–22 per episode. Drop an open 4B model into the same harness and it solves nothing: 0.0% on LIBERO.

So can we post-train a small open model into a robot coding agent, the same way coding and tool-use agents are trained?

We’re excited to share CaPtain, an end-to-end pipeline for post-training Code-as-Policy robot agents.

The agent writes code against 22 low-level primitives (perceive, measure, move, check), all the way down to IK-level motion. There are no grasp() skills and no VLA underneath. We teach it with SFT, then sharpen it with on-policy RL using a critic that reads the simulator’s physics state, information the policy never sees.

The results:

Qwen3.5-4B on held-out LIBERO goes 0.0% → 55.4% (SFT) → 73.8% (RL). RL adds +17 to +21 points across LIBERO-PRO, LIBERO-Plus and LIBERO-MAX.
RL ran only on LIBERO, and the model carries over to other benchmarks with no further training: RoboTwin 2.0 Hard 22.6% (above X-VLA’s 20.9%), SimplerEnv WidowX 83.3%, CaP-Bench 75.0, plus RoboDojo.
The same API and the same model drive a real arm with no real-world data.

The craziest part: none of what the model learned is tied to an action space. It learned to look, measure, move and check, so the embodiment lives below the API.

Everything is open under MIT: weights, harness, SFT data, RL code and evals. It’s already at ~1.5K downloads on HF 🔥

We’re on HF Daily Papers today, so an upvote would mean a lot 🙏

📄 Paper: https://t.co/uomNje4xVB
💻 Code: https://t.co/61TTD3qCim
✍️ Blog: https://t.co/G74LlwzMa0

How it works, what broke, and robot videos below 🧵 (1/N)

[NoaX @noahatwhere]

X 上的博士生：

PhD 焦虑 + 顶会 Deadline + OpenReview + Reviewer 2 + arXiv + H100 + CUDA OOM + Codex + Claude Code + Vibe Research + 一作执念 + 论文被拒 + Rebuttal + 熬夜跑实验 + 导师 Push + Funding 焦虑 + 学术泡沫 + Industry vs Academia + LeetCode + 量化转行 + 创业幻想 + 研究方向 Pivot + Paper 没写完 + GitHub 开源 + Google Scholar + 引用焦虑 + 咖啡续命 + Imposter Syndrome + 想 Dropout + 还想改变世界

[Yijia Fan @YijiaFanCSAI]

@AnselDias Yeah that is what we want

[Yijia Fan @YijiaFanCSAI]

🤖 Can a 4B model write code that drives a robot arm, using nothing but low-level IK?

Excited to share CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents. It’s on HF Daily Papers today, so an upvote or star would mean a lot 🙏

Why this matters

Code-as-Policy lets an LLM control a robot in closed loop by writing code instead of emitting raw actions. Frontier closed models have already shown that this works. On the open-source side, though, two things have been missing: a complete, reproducible post-training recipe, and a harness where a small model actually solves tasks through low-level IK alone, with no hand-crafted high-level skills. CaPtain fills both gaps.

The harness

🔹 Low-level IK only. There are no pre-built pick/place primitives, so the model has to decompose the task and express it in code itself.
🔹 ReAct loop. The model perceives the scene, writes code, executes it, and observes the result.
🔹 Self-correction. When execution errors or the task isn’t done, the model reads the feedback and revises its code.

Training (Qwen3.5-4B)

① SFT: we distill agent trajectories from GPT-6 Astra so the model learns how to operate inside the harness.
② RL: we run on-policy RL in simulation with a privileged critic that reads the simulator’s physical state to improve value estimation. The policy itself only sees standard observations, so nothing extra is needed at deployment.

Results

📈 LIBERO success rate goes from 0.0 for the base model to 55.4 after SFT and 73.8 after RL.
✅ Zero-shot deployment on a real robot.
✅ Evaluation support for RoboTwin and RoboDojo.

Fully open source

We’ve released the weights, the agent harness, the RL training code, and the evaluation suite. The model already has about 1.5K downloads on HF 🔥 Thank you all! Feel free to reproduce it, build on it, and tell us what breaks. If you find it useful, a repost would help a lot.

📄 Paper: https://t.co/uomNje4xVB
🤗 HF: https://t.co/Zhfq7XfGNl
💻 Code: https://t.co/61TTD3qCim

[Yijia Fan @YijiaFanCSAI]

Yes, there’s latency, but it’s already far lower than frontier-model agents like Astra since it’s a local 4B model, and we haven’t done any inference optimization yet. The model writes code, observes, and rewrites in a loop, so it’s not a reactive controller today. With standard speedups (quantization, speculative decoding, shorter code turns), we think it can get close to VLA-level responsiveness. Your approach (generate once with a big model, freeze the code) is great for repeated, stable tasks. They’re complementary: CaPtain can recover and adapt, then the code that works can be cached as a frozen skill.

[Yijia Fan @YijiaFanCSAI]

Every strong code-as-policy robot agent today runs on a closed frontier model. You can’t fine-tune it, you can’t put it on the robot, and it costs $8–22 per episode. Drop an open 4B model into the same harness and it solves nothing: 0.0% on LIBERO.

So can we post-train a small open model into a robot coding agent, the same way coding and tool-use agents are trained?

We’re excited to share CaPtain, an end-to-end pipeline for post-training Code-as-Policy robot agents.

The agent writes code against 22 low-level primitives (perceive, measure, move, check), all the way down to IK-level motion. There are no grasp() skills and no VLA underneath. We teach it with SFT, then sharpen it with on-policy RL using a critic that reads the simulator’s physics state, information the policy never sees.

The results:

Qwen3.5-4B on held-out LIBERO goes 0.0% → 55.4% (SFT) → 73.8% (RL). RL adds +17 to +21 points across LIBERO-PRO, LIBERO-Plus and LIBERO-MAX.
RL ran only on LIBERO, and the model carries over to other benchmarks with no further training: RoboTwin 2.0 Hard 22.6% (above X-VLA’s 20.9%), SimplerEnv WidowX 83.3%, CaP-Bench 75.0, plus RoboDojo.
The same API and the same model drive a real arm with no real-world data.

The craziest part: none of what the model learned is tied to an action space. It learned to look, measure, move and check, so the embodiment lives below the API.

Everything is open under MIT: weights, harness, SFT data, RL code and evals. It’s already at ~1.5K downloads on HF 🔥

We’re on HF Daily Papers today, so an upvote would mean a lot 🙏

📄 Paper: https://t.co/uomNje4xVB
💻 Code: https://t.co/61TTD3qCim
✍️ Blog: https://t.co/G74LlwzMa0

How it works, what broke, and robot videos below 🧵 (1/N)

[Yijia Fan @YijiaFanCSAI]

Exactly where it gets tricky. Pure binary success is too sparse for a 4B model, so during RL a privileged critic reads the simulator’s physics state (which object is held, where it is relative to the target, whether it was dropped) and gives progress-aware credit. Grabbing the right cube but missing the container scores higher than grabbing the wrong one, but still well below success. The critic only exists at train time; the policy never sees privileged state at test.

[Yijia Fan @YijiaFanCSAI]

Thanks! Both mattered, in different ways. SFT does the heavy lifting from zero: base Qwen3.5-4B scores 0 on LIBERO and SFT brings it to 55.4, because the model has to learn the API and how to write closed-loop code at all. RL then takes it from 55.4 to 73.8, mostly by fixing physical failure modes like bad grasps, giving up early and calling finish too soon, which teacher traces don’t cover well.

[Yijia Fan @YijiaFanCSAI]

@kyiiiifan It's quite obvious that SFT is only in its cold start phase; we're still focusing more on RL, and Guava isn't open source either.

[Yijia Fan @YijiaFanCSAI]

@saisurisetti_ Thanks LOL

[Yijia Fan @YijiaFanCSAI]

Real arm: “put the cube that has the color of the sky into the container that is not round.”

The agent has to resolve both references from the image before it acts. It uses the same API and the same receipts, with no real-world training data.

[Yijia Fan @YijiaFanCSAI]

7/N

What we believe now: robot agents should be post-trained like every other agent. Fix the harness, use SFT for the cold start, and RL for the rest.

Train your own model with our pipeline, or test yours in our harness:
🤗 https://t.co/xRU1TX8gDA
🎥 Full walkthrough: https://t.co/RZ7XuKwmOX

[Yijia Fan @YijiaFanCSAI]

6/N

Real arm: “put the cube that has the color of the sky into the container that is not round.”

The agent has to resolve both references from the image before it acts. It uses the same API and the same receipts, with no real-world training data.

[Yijia Fan @YijiaFanCSAI]

5/N

RL beats SFT across held-out LIBERO and all four perturbation suites. The gains are not memorization of training layouts.

One honest caveat: late in training the policy keeps solving tasks but forgets to call finish(). Stopping needs its own credit signal, and that’s next on our list.

[Yijia Fan @YijiaFanCSAI]

4/N

Three things that broke along the way:

1️⃣ The model stopped looking. After SFT, swapping the input image didn’t change the predicted pixels, because the model had memorized LIBERO’s fixed layouts. Regenerating the data from wide-layout starts fixed it. For embodied SFT, layout diversity beats data volume.

2️⃣ RL went nowhere. The updates were a random walk. We measured the gradient-noise scale at about 9.5K episodes, so a 256-episode update is mostly noise. Accumulating to 1,024 episodes per step fixed it. Measure B_noise before you tune anything else.

3️⃣ Credit lives where physics changes. One binary outcome over 20–60 requests is too sparse. A critic over simulator state (AUC 0.97) assigns the credit: 65% goes to moves and 1.3% to perception.

[Yijia Fan @YijiaFanCSAI]

3/N

Here is real code the trained 4B model wrote. It clicks pixels to ground the bowl and plate, measures their offset, carries the bowl by exactly that offset, then re-measures before calling finish():

“Center offset ≈ 1.7 cm, within the 2 cm tolerance.”

None of this is a hand-written skill. Spatial understanding reaches the motors through code.

[Yijia Fan @YijiaFanCSAI]

Why code? A VLM already understands space pretty well: the bowl is left of the plate, a drawer opens along its handle. What it lacks is a channel to the motors. Code is that channel.

Robot calls never raise errors. Each one returns a receipt instead:
“stopped: contact · moved 0.12 m · peak 6.2 N · jaw 8 → 1 cm”
A small model can’t recover from a failure it can’t see. Much of what RL improves is how well the policy reads these receipts.
(2/N)

[Yijia Fan @YijiaFanCSAI]

@Dakshagarwal76 Sft brings more cold start and formatting learning, but for the cancellation of critic, we will release it later this month (such as the variant of pure GRPO). Thank you for your attention!

[Yijia Fan @YijiaFanCSAI]

What if the future of robot intelligence looks more like a coding agent than a traditional robot policy?

That's our bet with CaPtain.

See the world → reason → write code → execute → observe → correct.

We believe Code-as-Policy offers a path to scale robotic intelligence through visual reasoning and computation, not just more robot demonstrations.

Our 4B model is just a first step. 🚀

[Yijia Fan @YijiaFanCSAI]

🚀 Introducing CaPtain: Fully Open-Source Code-as-Policy Robot Agents with Just 4B Parameters!

Can a small VLM control real robots by writing Python, using only low-level primitives and IK — without relying on frontier models or pretrained VLA policies?

We made it work. And we're open-sourcing EVERYTHING. 🏴‍☠️

🤖 Qwen3.5-4B + SFT + On-policy RL
LIBERO: 0.0% → 55.4% → 73.8%
Zero-shot transfer to real robots
Generalizes to RoboTwin, RoboDojo & more
No high-level robot skills — just perception, Python, and low-level IK!

Our pipeline distills agentic reasoning and self-correction from GPT-6 Astra trajectories, then improves the policy through simulator-based RL with a privileged critic. At inference, the 4B policy operates without privileged simulator state.

🔥 Everything is OPEN: model weights, SFT data, agent harness, SFT/RL training code, and evaluation benchmarks.

We hope CaPtain makes reproducible, small-scale Code-as-Policy robotics accessible to everyone!

📄 Paper: https://t.co/p52OTv9JUI

💻 GitHub: https://t.co/61TTD3qCim

🤗 Model: https://t.co/T75ljn564Q

📦 Data: https://t.co/BufR5F7NYC

⭐ Stars, feedback, and contributions are welcome!

[Yijia Fan @YijiaFanCSAI]

A broader bet behind CaPtain:

We believe Code-as-Policy could be one of the most promising ways to scale visual reasoning into physical intelligence.

Rather than mapping pixels directly to actions, why not let models reason over visual observations, write executable programs, and improve through interaction?

The exciting part is that this connects robotics to the broader progress in VLM reasoning, coding, and RL.

Scaling robot intelligence may not require scaling robot action data alone.