Bingao Chen

Undergrad @Tsinghua_Uni | Intern @Stanford | Prev @nyuniversity

[Bingao Chen @bingao_chen]

Wrist cameras give robots close-up views, but only where the hand happens to be.
What if the policy could decide where to look?

Introducing FoveaVLA: a VLA that replaces the wrist camera with learned foveation actions.

FoveaVLA predicts where it will need to look next, then crops that region from the full-resolution external image. The full view gives context; the foveal crop gives close-up detail.

With one external camera and no wrist camera, FoveaVLA reaches 70.8% success on 48 real-world trials, vs. 52.1% for π0.5 with a wrist camera.

• No additional demonstrations
• No camera calibration
• No wrist camera

Paper, code, and models are all open 🧵

[Bingao Chen @bingao_chen]

When I tried using Astra to tackle RoboMME a month ago, the results were still below human performance. Now the benchmark has almost been solved. Great work @liu_yuejiang !

[Yuejiang Liu @liu_yuejiang]

What should a robot remember? Can it learn to remember better than we tell it to?

Excited to share MEMI: Reverse-Engineering What Robots Should Remember

📈 97.0% end-to-end, 99.4% w/ low-level controller on RoboMME, nearly solving it for the first time

🧵👇

[Chongkai Gao @GChongkai]

🤖Robots need world models for visual planning, but they don't need to imagine every frame.

Introducing DreamCan: Sparse Visual Planning for Long-Horizon Robot Manipulation.

Project: https://t.co/0yEO3e7XwR
Code: https://t.co/qqsP6bYxSg
Paper: https://t.co/HALcug4TXb

[Bingao Chen @bingao_chen]

@LihanZha @GuyTvt Huge thanks to @StanfordAILab and @StanfordHAI for the inspiring research environment and support. I'm truly grateful for the opportunity to learn from and collaborate with such amazing researchers!

[Bingao Chen @bingao_chen]

How it works. FoveaVLA builds on π0.5, and from one shared observation it outputs two things:

• an action chunk, from the flow-matching action expert
• a foveation action: one box per external view, decoded with PaliGemma's existing location tokens, so no new vocabulary is needed

Three details make it practical:

Reuses DROID annotations. Target boxes come from existing RoboInter interaction annotations: the smallest rectangle around the gripper, the object, and the target region.

It looks ahead. The box targets the region needed at the next query, because the scene moves while the current chunk executes.

No added wait. Box tokens are decoded while the robot executes the current chunk, so the crop is ready when the next frame arrives.

[Bingao Chen @bingao_chen]

Paper, code, and models are all open:

🌐 Project: https://t.co/aC8rOTn0M6
📄 Paper: https://t.co/UljVLkqCFj
💻 Code: https://t.co/Cz1Rt2b21a
🤗 Models: https://t.co/9waHBmY9pl

Many thanks to Lihan Zha(@LihanZha ), C. Karen Liu and Guy Tevet (@GuyTvt ) for their guidance throughout this project.

[Bingao Chen @bingao_chen]

@jiazhi_yang2024 Thank you Jiazhi!

[Jiazhi Yang @jiazhi_yang2024]

Robots need to learn where to look next.

FoveaVLA dynamically predicts bounding boxes to identify regions relevant to the policy's decisions.

This reminds me of 'visual search' in general VLMs (e.g., V*: https://t.co/juTBTE9IPd).

Such a capability could be potentially more critical for robots operating in large, cluttered workspaces, such as humanoids.

[Bingao Chen @bingao_chen]

@QianzhongChen Thanks Qianzhong!

[Qianzhong Chen @QianzhongChen]

Great work @bingao_chen

[Bingao Chen @bingao_chen]

@LihanZha Thanks so much for your guidance and support, Lihan! Really enjoyed working with you on this!

[Bingao Chen @bingao_chen]

@TX_Leo_Wang Thanks, Leo!

[01azPublic @silic2023]

Count the visual information, not just the cameras. FoveaVLA removes the wrist camera, but its single-camera setup uses stereo-derived depth alongside RGB and learned crops. In the paper's 48-trial test, removing depth drops success from 70.8% to 20.8%. That's a useful constraint for anyone trying to reproduce the hardware savings.

[Zhi (Leo) Wang @TX_Leo_Wang]

Great work @bingao_chen

[Lihan Zha @LihanZha]

Wrist cameras provide high-res views of manipulation, but they couple perception and control, suffer from motion blur, and introduce embodiment-specific viewpoints.

With FoveaVLA, we show that robots can do better without wrist cameras! Instead, we teach the VLA to anticipate where future interactions will happen and focus its high-res perception there. By jointly predicting actions and where to look next, the robot can proactively gather the visual details it needs for precise manipulation! See the cool results from @bingao_chen

[Bingao Chen @bingao_chen]

@yinghan__chen Thanks, Yinghan!

[Yinghan Chen @yinghan__chen]

An amazing work done by Bingao at TML, replacing the wrist camera with learned foveation actions!

[Bingao Chen @bingao_chen]

This is the trade-off VLAs face today.

A wrist camera gives local detail but limited context.

An external camera sees the whole scene, but resizing 1280×720 down to 224×224 blurs the gripper and the marker into a few pixels.

FoveaVLA keeps both: the full view for context, plus a learned crop at source resolution. In this rollout, that crop has 16.8× the resolution of the resized view.

[Bingao Chen @bingao_chen]

Wrist cameras give robots close-up views, but only where the hand happens to be.
What if the policy could decide where to look?

Introducing FoveaVLA: a VLA that replaces the wrist camera with learned foveation actions.

FoveaVLA predicts where it will need to look next, then crops that region from the full-resolution external image. The full view gives context; the foveal crop gives close-up detail.

With one external camera and no wrist camera, FoveaVLA reaches 70.8% success on 48 real-world trials, vs. 52.1% for π0.5 with a wrist camera.

• No additional demonstrations
• No camera calibration
• No wrist camera

Paper, code, and models are all open 🧵

[Bingao Chen @bingao_chen]

Same trial, same scene, same instruction: "Place the black marker in the yellow bowl."

π0.5 sees an external camera plus a wrist camera. The wrist view gives close-up detail but little context, and it picks up the blue marker.

FoveaVLA sees a single external camera. It crops in on the right marker, follows it to the bowl, and finishes the task.