Siyao

PhD student @ MMLab of NTU

[Junlin Han @han_junlin]

We live in a multimodal world. We see, talk, act, and dream.

Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch?

Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition

So what’s the path forward?

Introducing: Towards Physics of Multimodal Pretraining (https://t.co/xgfbdDPFq4)

We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

[purusha @purusa0x6c]

I was reading this paper

HumanCLAW: Can vision-language acts through a body?

In this benchmark, every model runs through the exact same closed loop, at a 0.5-second cadence, until it stops, succeeds, or exhausts the 100-step episode budget. the only thing that differs across the nine is the frozen VLM backbone plugged into the harness.

For every model, the same loop repeats every 0.5 s until it stops or hits the 100-step budget:

1) Look at the egocentric camera view + text history
2) Describe what it sees and where its body is
3) Plan a goal for the next few seconds
4) Pick one atomic skill with a parameter (walk, turn, step, climb, sit, stop)
5) A verifier rejects unsafe choices, then a motion generator turns the skill into 0.5 s of real body motion
6) The half-physics simulator applies gravity and collisions, and returns the next view

Score = three staged checks:
Did it (a) see the target, (b) get within 20 cm and stop, (c) sit on it? Each counts only if both the objective fact happened and the model acknowledged it. They also log collisions, objects knocked over, and token cost

Gemini-3.1 tops all three stages (64.9% Find, 42.4% Nav, 16.8% Interact), so its the best model in the set end to end, and it does it at lower token cost than the other frontier models.

This shows google is focusing heavily on the vision capabilities of AI models. recently, they released gemini robotics ER2, which also shows they are moving more and more toward physical intelligence.

The gemini series models are nowhere near OpenAI or Anthropic when it comes to coding. And as we can see, OpenAI and Anthropic aren't even focusing much on vision capabilities. their models are performing really poorly on this benchmark.

I think that, in the coming months, we are going to see major improvements in vision with gemini's upcoming models..maybe 30% or even more.

but I also think it isnt going to be easy. tracking and navigation are the easier parts, but interaction is something that requires self embodiment behavior within the model itself. for interaction, it needs to know where and how every part of its body is moving, how its interacting with objects under gravity, and many other physical constraints.

nonetheless, I am still excited about the future.

[Siyao @siyao94]

@FkSg16KN @HuggingPapers HumanCLAW is not scripted. Here, the human body ACTs freely in the world, and outcomes come from physics. This makes a big difference: when scripted, the VLM always plays in a scripted house, where everything is perfect and everything is pre-arranged.

[Siyao @siyao94]

@FkSg16KN @HuggingPapers Not exactly. Alfred is fully scripted. : no body, no physics, only floating camera, the object simply "snaps" into place. VirtualHome adds a human avatar, but every action just plays a pre-made animation, still no physics.

[Siyao @siyao94]

@weikaih04 @Kuvvius @liuziwei7 @ManlingLi_ @RanjayKrishna @LINJIEFUN @AmyZhaoMIT @chuan_guo92603 No explicit pose or proprioceptive state. The system prompt only tells the VLM that it is a blue humanoid robot seeing the world first-person. We also tell it how to locate itself in the view: its own feet, hands, and body appear at the bottom center of the frame.

[Siyao @siyao94]

@linhdoan02 @Kuvvius Exactly! That is just what our HumanCLAW wanted to step forward to.

[Siyao @siyao94]

@dsandzhiev @Kuvvius Just imagined the robot in the wild in the future would need more "reasoning" to be generalisable. After all the real world is so complicated that data to train policy may be not feasible to cover all.

[Siyao @siyao94]

@AdhamGhazali @Kuvvius That's true, for now. But just imagined when we really want robots to move into the open world, rather than staying on the tabletop, we may need more reasoning-based agents.

[Siyao @siyao94]

🧵Original post: https://t.co/ZTjLb3MkMJ

[Jiawei Gu @Kuvvius]

🦾HumanCLAW: Can Vision-Language Models Act Through a Body?

A VLM can spot the sofa in a second. But can it get a body there and sit down?

We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻https://t.co/9RDY1RO14q

[Siyao @siyao94]

Full-body control this smooth from @GoogleDeepMind is amazing to see! 🤖✨

We recently released a related work, HumanCLAW, exploring: given a physical body🦾, how intelligently can the "brain" (the VLM) **ACT** in the world?

📄 https://t.co/2kpzo4tn4t
🌐 https://t.co/khn1Uqmjj4

[Google DeepMind @GoogleDeepMind]

One brain. For any robot. 🤖

We’re launching Gemini Robotics 2: our next-generation physical AI bringing full body intelligence to humanoids, advanced dexterity, multi-robot teamwork and more.

[Siyao @siyao94]

@GoogleDeepMind Amazing! We recently did a related work, HumanCLAW, exploring: given a physical body, can the brain (VLM) ACT in the world?

Finding: today's VLMs lack self-awareness: they don't know where they are, or whether hit something.
📄 https://t.co/2kpzo4tn4t
🌐 https://t.co/khn1Uqmjj4

[Jiawei Gu @Kuvvius]

Robotics felt different this month. Google DeepMind and Anthropic made the same question hard to ignore: can a reasoning model decide what a whole body/robot should do next?

Gemini Robotics 2 put an embodied reasoning model above a VLA for whole-body control. Anthropic tested general-purpose reasoning models at several levels of control.

- https://t.co/2jQQp1HjyE
- https://t.co/ed0wZcIaRO

The bad news is clear. General-purpose reasoning models still cannot control a humanoid on their own. They need a pretrained policy underneath.

Still, I think this might actually work. A strong model (w/ excellent multimodal and reasoning capabilities) could really do it one day.

HumanCLAW is our attempt to get an early glimpse of that future.

[Jiawei Gu @Kuvvius]

🦾HumanCLAW: Can Vision-Language Models Act Through a Body?

A VLM can spot the sofa in a second. But can it get a body there and sit down?

We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻https://t.co/9RDY1RO14q

[Ziwei Liu @liuziwei7]

🦾Can Vision-Language Models Act Through a Body🦿

🤖HumanCLAW🤖 gives VLM a human body: atomic skills in, real physics out.

🤳Frontier VLM completes just 16.8%: what's missing is embodied *self-body-awareness*

- Project: https://t.co/lUhzSCG50i
- Paper: https://t.co/Igzjv2M7ld

[DailyPapers @HuggingPapers]

HumanCLAW: Can Vision-Language Models act through a body?

Meta and collaborators introduce a benchmark that decouples action decisions from motor control. Across 1,218 embodied episodes, no VLM solves it — the best reaches only 16.8%. Models lack embodied self-awareness: they lose track of the body they control.

[Siyao @siyao94]

@HuggingPapers Thanks @HuggingPapers for sharing our new work!

[Siyao @siyao94]

@Michael_J_Black Congratulations again, Michael!!! Wish all the best to your new advanture!

[Jianyang Gao @gaoj0017]

The TurboQuant paper (ICLR 2026) contains serious issues in how it describes RaBitQ, including incorrect technical claims and misleading theory/experiment comparisons.

We flagged these issues to the authors before submission. They acknowledged them, but chose not to fix them. The paper was later accepted and widely promoted by Google, reaching tens of millions of views.

We’re speaking up now because once a misleading narrative spreads, it becomes much harder to correct. We’ve written a public comment on openreview (https://t.co/nDVjmNhATM).

We would greatly appreciate your attention and help in sharing it.

[Siyao @siyao94]

The weekend of Singapore PhD student.

[Le Chen @clthegoat]

Learning to play the piano with two robot hands is super challenging, even in simulation! It requires coping with bimanual coordination at high speed to achieve human-level dexterity. We introduce RP1M, a large-scale robot piano-playing motion dataset, featuring ~1M trajectories over 2k music pieces.

Website: https://t.co/Bvk90MV6KH
Paper: https://t.co/45j6y1NncI