Kaiyuan Eric Chen ✈️RSS🇦🇺

System + Robotics Phd & Postdoc @UCBerkeley | @UCLA Alumni

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

Graph-as-Policy (GaP) is a new variant of Agentic Robotics from @NVIDIA and @UCBerkeley that builds computation graphs (like ROS) to ensure modularity, manage complexity, and facilitate interpretability.
🧵Open code and paper: https://t.co/4UfdBh3DQK
#Robotics #Automation #AgenticAI #EmbodiedAI

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

I will be presenting RoboVista at #RSS2026 today! This joint work with @GoogleDeepMind assess how effectively current VLMs process diverse robotic applications (esp. field robotics!) . Fun fact: most of all annotators already have PhD degrees. Come talk to me at poster session!

[Shuangyu Xie @Shuangyu0_0Xie]

🤖How well do today's VLMs actually understand real-world robotics? 👀

Excited to share RoboVista at #RSS2026 — a systematic evaluation and benchmark for VLMs across diverse, real-world robot applications. Website, dataset and paper: https://t.co/saNSk3iVsg

Developed by researchers at @UCBerkeley, @GoogleDeepMind, and @Princeton. 🧵👇

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

Thanks to the great team !
including: @UksangYoo @ruwantw from @BoschGlobal
@guanzhi_wang @Adithya_Murali_ @balakumar_ @yukez @DrJimFan #spencer_huang from @nvidia
@Shuangyu0_0Xie @letian_fu @uynitsuj @talking_kim @sandeep_b24 @HaoruXue from and @Ken_Goldberg from @UCBerkeley

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

Graph-as-Policy (GaP) is a new variant of Agentic Robotics from @NVIDIA and @UCBerkeley that builds computation graphs (like ROS) to ensure modularity, manage complexity, and facilitate interpretability.
🧵Open code and paper: https://t.co/4UfdBh3DQK
#Robotics #Automation #AgenticAI #EmbodiedAI

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

The paper, more demo videos, 8 benchmarks, the skill library, and the code are all open. 🔓
🌐 Site: https://t.co/TpnjytJ2KR
📄 Paper: https://t.co/XHA7bDxBDT

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

The Results:
📦 Under object pose variation: In benchmarks where model-free VLA policies dropped to 20% success rate, GaP achieved 93%-99% success rates.
🧼 Automated fine- tuning: For a bimanual crate-washing task, GaP matched performance of hand-engineered code.

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

How GaP works:
🗣️ Task Description: Describe a task (e.g., "make popcorn - repeat"), workcell, object and pose range, and provide a set of model-based and model-free skills.
🧠 Multi-Agent Orchestration: A multi-agent harness, built over coding tools like #Claude and #Gemini, decomposes a special prompt and assembles an interpretable computation graph from an open skill library
🔄 Self-Learning in Sim: The graph is rehearsed in simulation, using contact feedback to diagnose its own failures, and rewrites its own structure until performance plateaus.
🦾 Sim-to-Real: The optimized graph is then exported to run on the physical robot.
GaP can provide the interpretability and reliability of classical engineering, but with coding agents doing the authoring and fine tuning. 🤯

[Zhengzhong Tu @_vztu]

📍 𝗖𝗮𝗻 𝗔𝗜 𝗡𝗮𝘃𝗶𝗴𝗮𝘁𝗲 𝗠𝗮𝗽𝘀 𝗟𝗶𝗸𝗲 𝗛𝘂𝗺𝗮𝗻𝘀 𝗗𝗼? 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵! 🗺️🤖

𝘙𝘦𝘢𝘥𝘪𝘯𝘨 𝘮𝘢𝘱𝘴, like Google Maps and Theme Park Maps, is second nature for humans. It is a highly challenging task that requires visual understanding, spatial reasoning, and long-horizon planning. We're curious - 𝗖𝗮𝗻 𝗟𝗮𝗿𝗴𝗲 𝗩𝗶𝘀𝗶𝗼𝗻-𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗩𝗟𝗠𝘀) 𝗱𝗼 𝗶𝘁 𝘁𝗼𝗼? 🤔

We’re excited to share 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵, the first-ever dataset and benchmark specifically designed for evaluating how well LVLMs perform on pixel-based map navigation tasks! 🚀

🔑 𝗪𝗵𝘆 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵 𝗶𝘀 𝗮 𝗚𝗮𝗺𝗲-𝗖𝗵𝗮𝗻𝗴𝗲𝗿:
• 📌 1600+ Complex Pathfinding Queries from 100 uniquely challenging map scenarios (urban areas, theme parks, universities, malls, and more).
• 📌 Introduces Map Space Scene Graph (MSSG): a novel data structure for mapping visual landmarks and spatial relationships to structured navigation tasks.
• 📌 Evaluates state-of-the-art LVLMs like GPT-4o, Llama-3.2, and Qwen-2-VL under zero-shot and Chain-of-Thought (CoT) reasoning methods, revealing key insights into their spatial reasoning and navigation abilities.

🚩 𝗞𝗲𝘆 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀:
• Despite their impressive capabilities, current LVLMs struggle significantly with spatial reasoning and structured decision-making.
• CoT prompting boosts spatial reasoning performance but sometimes introduces redundant details.

👀 𝗖𝗵𝗲𝗰𝗸 𝗼𝘂𝘁 𝗼𝘂𝗿 𝗳𝗶𝗻𝗱𝗶𝗻𝗴𝘀, 𝗱𝗮𝘁𝗮𝘀𝗲𝘁, 𝗮𝗻𝗱 𝗰𝗼𝗱𝗲 𝗵𝗲𝗿𝗲:
🔗 Arxiv: https://t.co/41aeScvzrb

Huge thanks to our incredible collaborators for making this happen, from @TAMU, @UCBerkeley, @mbzuai, @UMich, and @UCRiverside! 🎉

Let’s continue to bridge the gap between human intuition and AI navigation! 🗺️💡

[SkyPilot @skypilot_org]

Releasing a guide for scaling LLM embedding generation by 9x compared to autoscaling cloud services:

⚡️9x more GPUs ➡️ 9x speedup
💰61% cost reduction with spot

The secret? Going multi-region with @skypilot_org to overcome single-region capacity limits.
https://t.co/GFXmxCSrm1

[SkyPilot @skypilot_org]

New model: Spin up Gemma 3 in your cluster/cloud

$ sky launch gemma3.yaml

https://t.co/pyGb9PcaMT

[SkyPilot @skypilot_org]

Try our live demo for the RAG on legal documents: https://t.co/rfeWUonabO

[SkyPilot @skypilot_org]

💡We built an open-source RAG with DeepSeek-R1!

Here's what we learned:
📄 Don’t use DeepSeek R1 for retrieval; Use specialized embeddings — Qwen’s embedding model is amazing! @Alibaba_Qwen
🤖 Do use R1 for response generation @deepseek_ai
🔧 Use AI infra, vLLM @vllm_project & SkyPilot, to boost performance by 5x & scale up by 100x!

Check out our complete code and learnings: https://t.co/mNBKV0Nlec

[SkyPilot @skypilot_org]

🚀We built a million-scale semantic image search with VectorDB and OpenAI’s CLIP. 🖼️

This is achieved by scaling out image embedding generation using SkyPilot — cutting time from ⏲️120 hours to 1 hour and costs from 💸$231 to $46!

See how: https://t.co/DtWMzJoBS2

Great thanks to the fantastic tools from: @trychroma @huggingface
@OpenAI

[SkyPilot @skypilot_org]

SkyPilot now supports Janus-Pro by DeepSeek!

• Text to image generation ✍️🖼️
• Image Q&A 📸🤖

On any of your infra!

https://t.co/THiZtljlbp

[Kaiyuan Eric Chen ✈️RSS🇦🇺 @keplerccccc]

@zoodotdev @apgoldberg1