Suhwan Choi

Aiming to imitate human artificially in terms of multimodal agent

[Suhwan Choi @milkclouds00]

@sirwart I recommend having a look at my multi-stage dataset format! I bet this is interesting one to be referred to.

https://t.co/m6dWQXVIp2

Also since file-per-episode requiring large numbers of file which is harmful for common distributed storage, I’m considering adopting webdataset

[Quanting Xie @DanielXieee]

While everyone’s betting on world action models, UMI + scaling just quietly solved it. No embodiment gap. Just works.

[Ai2 @allen_ai]

📢 Introducing vla-evaluation-harness—a unified, fully open framework to evaluate any VLA model on any robot simulation benchmark.

Integrate your model once. Integrate the benchmark once. The full cross-evaluation matrix fills itself. 🧵

[Suhwan Choi @milkclouds00]

The comparison seems to be quite unfair, (1) training IDM on world model when robotics world model is given by NVIDIA(cosmos-predict2) and (2) training VLA from MLLM is by far more advantageous for former one. Training cost of “robotics” world model(which is HUGE) is neglected, but baseline VLA train is started from pure MLLM(PaliGemma). It would be much better if they included more ablation for base video world models which is not trained on robotics domain.

[Suhwan Choi @milkclouds00]

Reimplemented NVIDIA's VLA-0 using TRL SFTTrainer.

While common VLA codebases are over 10,000 lines, vla0-trl contains only ~1,200 lines. Gets ~90% on LIBERO by fine-tuning Qwen2-VL to predict actions as text.

Good starting point if you want to build your own VLA.

https://t.co/bF3YdvfMzi

[Suhwan Choi @milkclouds00]

@jadechoghari @OpenAI This is awesome but after I check the docs I proposed more modernized EnvHub environment repository design: https://t.co/y8ZN41jex2

Happy to accept any feedback.

[Suhwan Choi @milkclouds00]

@TairanHe99 @GeneralistAI Since success rate can be decomposed into (sort of) multiplication of validation loss for single action we can expect similar trend, but some pitfalls(e.g. emergent performance) can occur. https://t.co/ea9k8Ij1Jk here you can see some related contents.

[Suhwan Choi @milkclouds00]

@ArmenAgha Which encoding using for multimodal data? Image encoding or video encoding?

[Suhwan Choi @milkclouds00]

https://t.co/wBogV2QIly

"it's much easier to train a 'blind gymnast' than a robot that sees and manipulates" — so true!

So we should pretrain on pixel-to-action, and we propose desktop interface is the key to scale. We collected 1.3k hours of screen(monitor)-keyboard-mouse data within a week and showed this data improves robotics policy.

If you’re curious, here’s our latest work:

[Suhwan Choi @milkclouds00]

🚀 Rethinking agent learning: Vision might be the next language.

What if AI learned to act not from text, but from what it sees—just like humans?

In the world of AI, we're constantly rethinking how agents should perceive and act. Recently, Deepseek-OCR made headlines by showing that vision tokens can compress text tokens by a factor of ten. This breakthrough is more than just an advance in OCR—it's a fresh signal that maybe, future models can understand as much (or more) from vision as they do from text. Even leaders like Andrej Karpathy are asking: is text tokenization truly the best route?

As a researcher, I've been thinking a lot about this, especially for robotics and general embodied AI. Collecting real-world human action data for pretraining is difficult: robots have disparate control interfaces and gathering data at scale is costly.

So, what if we leverage what billions of people already have at their fingertips—the keyboard and mouse? In my latest project, D2E, we've built a fully automated pipeline to collect large-scale human-action data from desktop interactions.

◆ The result? One supervisor gathered 300 hours of data in just a month, and our web pipeline collected 1,000 hours of video in a single week—orders of magnitude more efficiently than prior datasets.
◆ All this without expensive robots or manual language annotation.

Right now, most VLA models depend on explicit language instructions. But I believe that, as with humans, future agents should learn to "see" and interpret instructions visually—through UI cues or game objectives—rather than as raw text tokens. Imagine agents that can look at the screen, read "Capture the objective," and act, just as people do.
If you’re interested in building more human-like, visually grounded AI agents—or want to explore the D2E dataset and tools—check out our project page: https://t.co/ibG32MyCYI

I welcome feedback, discussion, and collaboration!

[Suhwan Choi @milkclouds00]

🚀 Rethinking agent learning: Vision might be the next language.

What if AI learned to act not from text, but from what it sees—just like humans?

In the world of AI, we're constantly rethinking how agents should perceive and act. Recently, Deepseek-OCR made headlines by showing that vision tokens can compress text tokens by a factor of ten. This breakthrough is more than just an advance in OCR—it's a fresh signal that maybe, future models can understand as much (or more) from vision as they do from text. Even leaders like Andrej Karpathy are asking: is text tokenization truly the best route?

As a researcher, I've been thinking a lot about this, especially for robotics and general embodied AI. Collecting real-world human action data for pretraining is difficult: robots have disparate control interfaces and gathering data at scale is costly.

So, what if we leverage what billions of people already have at their fingertips—the keyboard and mouse? In my latest project, D2E, we've built a fully automated pipeline to collect large-scale human-action data from desktop interactions.

◆ The result? One supervisor gathered 300 hours of data in just a month, and our web pipeline collected 1,000 hours of video in a single week—orders of magnitude more efficiently than prior datasets.
◆ All this without expensive robots or manual language annotation.

Right now, most VLA models depend on explicit language instructions. But I believe that, as with humans, future agents should learn to "see" and interpret instructions visually—through UI cues or game objectives—rather than as raw text tokens. Imagine agents that can look at the screen, read "Capture the objective," and act, just as people do.
If you’re interested in building more human-like, visually grounded AI agents—or want to explore the D2E dataset and tools—check out our project page: https://t.co/ibG32MyCYI

I welcome feedback, discussion, and collaboration!

[AK @_akhaliq]

D2E

Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI

[Andy Zeng @andyzengineer]

"Train a model to predict how many timesteps left until task success"

- a simple, yet powerful way to get rewards from episodic BC data

Lots of nuggets in the paper (including steps-to-go fn is distributionally multimodal)
Kamyar's post 👇 on how it drives RL self-improvement

[Ernest Ryu @ErnestRyu]

최근 서울대 교수 56명이 해외 대학으로 ‘이탈’하였다는 기사와 함께 서울대의 인재유출이 우려할만하다는 여론이 형성되고 있습니다. 그런데 저는 기사들의 초점이 서울대 교수들의 부족한 연봉에만 있는 것이 조금 아쉽습니다. 저는 근본적인 문제가 조금 다른 포인트에 있다고 생각합니다. (1/N)

[Suhwan Choi @milkclouds00]

Augment Code enabled me to write 10000 lines of code during single weekends!
https://t.co/3ceqmuYEON

I used Augment Code to
1. write a PEP-like standard markdown file which states why, what, how to implement
2. argue with me
3. After confirm, implement the document to codebase

[Augment Code @augmentcode]

Remote Agent is live in VS Code (v0.472.1).

Early users are already shipping while they sleep. One dev launched 3 agents before logging off - woke up to a PR, docs, and a fixed flaky test.

We’re celebrating wins like that by giving 5 devs 4 months of Augment Max ($1K in value each).

RT + reply with your worst backlog task. Every 100 RTs = one more winner

[Augment Code @augmentcode]

Remote Agent is live in VS Code (v0.472.1).

Early users are already shipping while they sleep. One dev launched 3 agents before logging off - woke up to a PR, docs, and a fixed flaky test.

We’re celebrating wins like that by giving 5 devs 4 months of Augment Max ($1K in value each).

RT + reply with your worst backlog task. Every 100 RTs = one more winner

[Suhwan Choi @milkclouds00]

@Kangwook_Lee We may revisit @cloneofsimo ‘s solution here to explain “surprising” limitation of LLMs: https://t.co/CCT4dvgCoH

[Suhwan Choi @milkclouds00]

@giffmana We're building agent with harness. First of all, we provides data format for desktop usage and high-performance keyboard/mouse/screen recorder for desktop. It's the start point and must be start point.

For who have a interest, have a look.

https://t.co/eM7XyXhKh7

[Jiayi Pan @jiayi_pirate]

Quick ablations on CountDown:
Base model quality is key:

We run Qwen-2.5-Base 0.5B, 1.5B, 3B to 7B. 0.5B guess a solution and stop. From 1.5B, the model start learning to search, to self-verify and to revise its solutions, enabling them to achieve much higher scores.

[Andrej Karpathy @karpathy]

# RLHF is just barely RL

Reinforcement Learning from Human Feedback (RLHF) is the third (and last) major stage of training an LLM, after pretraining and supervised finetuning (SFT). My rant on RLHF is that it is just barely RL, in a way that I think is not too widely appreciated. RL is powerful. RLHF is not. Let's take a look at the example of AlphaGo. AlphaGo was trained with actual RL. The computer played games of Go and trained on rollouts that maximized the reward function (winning the game), eventually surpassing the best human players at Go. AlphaGo was not trained with RLHF. If it were, it would not have worked nearly as well.

What would it look like to train AlphaGo with RLHF? Well first, you'd give human labelers two board states from Go, and ask them which one they like better:

Then you'd collect say 100,000 comparisons like this, and you'd train a "Reward Model" (RM) neural network to imitate this human "vibe check" of the board state. You'd train it to agree with the human judgement on average. Once we have a Reward Model vibe check, you run RL with respect to it, learning to play the moves that lead to good vibes. Clearly, this would not have led anywhere too interesting in Go. There are two fundamental, separate reasons for this:

1. The vibes could be misleading - this is not the actual reward (winning the game). This is a crappy proxy objective. But much worse,
2. You'd find that your RL optimization goes off rails as it quickly discovers board states that are adversarial examples to the Reward Model. Remember the RM is a massive neural net with billions of parameters imitating the vibe. There are board states are "out of distribution" to its training data, which are not actually good states, yet by chance they get a very high reward from the RM.

For the exact same reasons, sometimes I'm a bit surprised RLHF works for LLMs at all. The RM we train for LLMs is just a vibe check in the exact same way. It gives high scores to the kinds of assistant responses that human raters statistically seem to like. It's not the "actual" objective of correctly solving problems, it's a proxy objective of what looks good to humans. Second, you can't even run RLHF for too long because your model quickly learns to respond in ways that game the reward model. These predictions can look really weird, e.g. you'll see that your LLM Assistant starts to respond with something non-sensical like "The the the the the the" to many prompts. Which looks ridiculous to you but then you look at the RM vibe check and see that for some reason the RM thinks these look excellent. Your LLM found an adversarial example. It's out of domain w.r.t. the RM's training data, in an undefined territory. Yes you can mitigate this by repeatedly adding these specific examples into the training set, but you'll find other adversarial examples next time around. For this reason, you can't even run RLHF for too many steps of optimization. You do a few hundred/thousand steps and then you have to call it because your optimization will start to game the RM. This is not RL like AlphaGo was.

And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch. A good example is a prompt like "Generate a poem about paperclips" or something like that. An average human labeler will struggle to write a good poem from scratch as an SFT example, but they could select a good looking poem given a few candidates. So RLHF is a kind of way to benefit from this gap of "easiness" of human supervision. There's a few other reasons, e.g. RLHF is also helpful in mitigating hallucinations because if the RM is a strong enough model to catch the LLM making stuff up during training, it can learn to penalize this with a low reward, teaching the model an aversion to risking factual knowledge when it's not sure. But a satisfying treatment of hallucinations and their mitigations is a whole different post so I digress. All to say that RLHF *is* net useful, but it's not RL.

No production-grade *actual* RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale. And intuitively, this is because getting actual rewards (i.e. the equivalent of win the game) is really difficult in the open-ended problem solving tasks. It's all fun and games in a closed, game-like environment like Go where the dynamics are constrained and the reward function is cheap to evaluate and impossible to game. But how do you give an objective reward for summarizing an article? Or answering a slightly ambiguous question about some pip install issue? Or telling a joke? Or re-writing some Java code to Python? Going towards this is not in principle impossible but it's also not trivial and it requires some creative thinking. But whoever convincingly cracks this problem will be able to run actual RL. The kind of RL that led to AlphaGo beating humans in Go. Except this LLM would have a real shot of beating humans in open-domain problem solving.

[Suhwan Choi @milkclouds00]

R1's key insight is simple. ORM with minimal guidance(answer/format) is scalable.

And it's normal to adopt scalable things(ORM) before non-scalable things(PRM/MCTS/...).

Future AI's training recipe may be: pretraining-ORM/SFT-PRM/MCTS

[Suhwan Choi @milkclouds00]

My guess is, o1 == R1 and o3 == o1 + MCTS
o3's unexpectedly high inference price(>$1K) is explainable with MCTS

https://t.co/C0IDmEvr5y

[Nat McAleese @__nmca__]

o1 was the first large reasoning model — as we outlined in the original “Learning to Reason” blog, it’s “just” an LLM trained with RL. o3 is powered by further scaling up RL beyond o1, and the strength of the resulting model the resulting model is very, very impressive. (2/n)