karthikpythireddi

MS EE Robotics & AI @stanford | all views are personal and not related to any firm neither an endorsement.

[LeRobot @LeRobotHF]

You don't need hardware to play with robots!

Thanks to this project you can have SO-ARM101 running an ACT policy inside your browser. Really cool project!

[Marco Pavone @drmapavone]

How much time should robots spend thinking?

Vision-Language Models are increasingly used as high-level planners for robots, and the prevailing strategy has been to scale test-time compute to boost capability. But more reasoning steps, bigger models, and longer memory all come with increased latency, tokens, and FLOPs—often with diminishing and uneven returns.

So when, and where, is test-time compute actually worth its cost? 🧐

We study three dominant scaling axes and find that each unlocks a distinct capability, showing that test-time compute is not a uniform lever:

- Chain-of-thought depth helps with tasks involving implicit semantic, physical, or spatial constraints, but its additional latency is not always necessary (on VLABench, a non-CoT model matches a CoT model on 44% of tasks).

- Model size governs the breadth of skills a planner can reliably draw upon, but its benefits appear only when those additional skills are actually required.

- Memory history improves performance on long-horizon, history-dependent tasks, but can actively hurt performance elsewhere.

Across all three axes, a consistent pattern emerges: the gap between cheap and expensive configurations is large, but highly non-uniform and task-dependent.

DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs) is a lightweight router that reads scene + instruction context and sends each task to the cheapest planner that can still solve it, allocating compute per task rather than committing to one fixed model.

👉 Takeaway: smart allocation of test-time compute can recover frontier-level planning at a fraction of the cost.

📄 Paper: https://t.co/H11V7q4zGj
🔗 Website: https://t.co/Es9RJaXE0o

Work led by @_jadelynn @milanganai

With an outstanding team of collaborators: @ajaysridhar0 @Mozhgan_nasr @katielulula Clark Barrett @jiajunwu_cs @chelseabfinn

#Robotics #VLM #EmbodiedAI #MachineLearning #TestTimeCompute

[Cristóbal Eyzaguirre Ercilla @CristbalEyzagu2]

1/ The biggest problem in video understanding today isn't the models. It's that we can barely run them.

Introducing StateKV: an inference-time method that makes pretrained video VLMs scale linearly with video length.🧵

🔗 https://t.co/VTnCZd7vdI

[NVIDIA @nvidia]

Two frontier labs. One accelerated computing platform. Congrats to @SpaceX and @AnthropicAI on the new compute partnership, powered by 220,000+ NVIDIA GPUs inside Colossus 1. The future of AI runs on NVIDIA.

[Percy Liang @percyliang]

I find myself repeatedly explaining the difference between open-weight (DeepSeek), open-source (Olmo), open-development (Marin). Let's see if this restaurant analogy helps:
- Open-weight: food is made behind closed doors, server brings you the dish
- Open-source: food is made behind closed doors, server brings you the dish and the recipe
- Open-development: you see the chef make the dish in the kitchen (and can shout suggestions while its cooking)!

[Hazel Nam @ ICML @hazel_heejeong]

The recording is out! Find our talk on stanford CS25 youtube📺

Also Causal JEPA will be at ICML. Looking forward to see you all in Seoul 😛

[Junfan Zhu 朱俊帆 ✈️ SIGGRAPH @junfanzhu98]

🤖🍇 Robotics & World Model Reading Club 06 Recap—SF0502 @saturdayrobotic
Video World Models as Simulator & Policy—Keynote from @tongzhou_mu (@RhodaAI), cohost @junfanzhu98, @aurorafeng_01
Robotics is no longer about learning actions—it’s about selecting actions from predicted futures.

🎬 Two Roles of Video World Models
- Simulator → learn physics from data, generate experience
- Policy → drive decisions via video-conditioned action generation
Goal: inject physical common sense from web-scale video into control.

🧪 I. Video Models as Learned Simulators
1) Data Synthesis (DreamGen, GR00T)
Pipeline = RAG + rollout + IDM labeling
- Prompt → retrieve similar robot videos → generate new task rollouts
- Inverse Dynamics Model (IDM) → pseudo actions → train policy
✅ Cheap, scalable, safe edge cases
❌ Open-loop → hallucination + error accumulation
⚠️ Insight: IDM is NOT the bottleneck
→ inverse mapping is easy; forward world modeling is hard
→ works with teleop / eval / random data; generalizes across robots

2) Inference-Time Planning (V-JEPA2)
- Action-conditioned video model
-Sample action sequences → rollout in latent → score vs goal
- Replan iteratively (receding horizon)
✅ Test-time scaling (more samples = better plans)
❌ Heavy compute vs real-time control
👉 Pipeline: Policy eval → prune → planning

3) Policy Evaluation (Veo, Ctrl-World)
- Use video model as simulator for scoring trajectories
- Acts as action filter / value proxy
✅ Unlimited rollouts (vs traditional sim limits)
❌ Less accurate than physics engines
🚨 Not real-time → offline selection before planning

🤖 II. Video Models as Policy
4 Paradigms
(1) Joint Video + Action Generation (DreamZero, GR1/2)
- Diffusion / flow matching over video+action
- Shared denoising → cross-modal reasoning
⚠️ Open: pretrained video ≠ pretrained for action

(2) Representation → Action (VPP, Video Policy)
- Video model = feature extractor
- Small diffusion policy = action head
⚡ Fast inference
⚠️ Partial denoising = control authority allocation
- none → action head decides
- full → video dominates
- partial → shared decision boundary

(3) Open-loop Generation (UniPi)
Generate full future video → IDM → actions
✅ Uses video model as-is
❌ Plan fixed → no reaction → brittle

(4) Closed-loop Generation (DVA, LingBot)
Generate → act → replace with real observation → repeat
✅ Continuous grounding → avoids hallucination
❌ Requires causal modeling + heavy infra
🔥 Core insight:
Video model ≠ decision maker
→ it proposes futures
→ policy = selecting among futures via translation/scoring

🧠 System-Level Truths
- Failure ≠ video problem → usually translation / constraint / IDM issue
- Action space is task-dependent (position vs others)
- Closed-loop = continuous alignment with reality

⚙️ Deployment Reality
Infra > model tweaks (latency, kernel fusion, precision)
100 robots ↔ 1000 GPUs? edge vs cloud tradeoff
Data unclear: UMI / egocentric promising but not converged
Perception bottleneck: camera latency, resolution, depth

⚠️ Fundamental Tensions
Latent vs Pixel
→ latent efficient but may drop task-critical info
→ representation choice caps capability
RL Warning
→ learned simulator ≠ ground truth
→ RL will exploit model bias (simulator hacking)
Tactile vs Vision
→ easy to add, but no internet-scale data → video dominates

🚀 Emerging Directions
- Diffusion distillation → faster generation
- Auto-regressive diffusion transformers
- Video models as simulator + policy + value function + data engine
👉 A unified computational primitive

🧩 Video world models are not just better perception—they redefine control:
From “predict → act” → to “generate futures → select actions.”
The bottleneck has shifted:
❌ Not model capability
✅ Grounding decisions into real-world constraints

[Junfan Zhu 朱俊帆 ✈️ SIGGRAPH @junfanzhu98]

https://t.co/TwJKaiX8l4

[Fei-Fei Li @drfeifei]

It’s 11th year and counting! Teaching the first lecture of @cs231n every year has been a highlight of my spring seasons. As usual, I asked students which departments or schools they come from @Stanford . Increasingly, students raise their hands to indicate that they come from all seven schools on campus, from @StanfordEng to @StanfordMed @StanfordHumSci @StanfordGSB @StanfordLaw @StanfordEd @stanforddoerr . AI is truly a horizontal technology that excites students across all backgrounds and disciplines!🤩

[Steven Feng @stevenyfeng]

We’re bringing back Stanford’s CS25 Transformers course tomorrow! 🤖

It’s open to everyone (in-person + online). Weekly talks (every Thursday) from top AI researchers.

One of Stanford’s most popular AI seminar courses. Don’t miss out!

More info below 👇 (1/7)

[Marco Pavone @drmapavone]

A central challenge in #physical #AI is data scarcity: vision-language-action (#VLA) models are fundamentally limited by the availability of high-quality robotics demonstrations.

In our recent work, we introduce R&B-EnCoRe (https://t.co/S34vtBELE1), a framework that enables models to self-bootstrap embodied #reasoning by leveraging synthetic visuo-textual data together with limited embodiment-specific experience. In essence, R&B-EnCoRe allows models to learn how to reason in an embodied setting.

Our approach treats reasoning as a latent variable and uses self-supervised refinement to learn reasoning strategies that are directly predictive of successful control—without human annotations, reward engineering, or external verifiers.

We validate the approach across a range of embodiments—including manipulation, navigation, and autonomous driving—and across model scales from 1B to 30B parameters, observing consistent improvements:
💪 +28% task success in real-world manipulation
🦿 +101% score in legged locomotion navigation
🚗 −21% collision rate in autonomous driving

Overall, this work highlights a promising direction: aligning internet-scale priors with embodiment-specific data to enable scalable, self-improving physical intelligence.

Kudos to an amazing team: Milan Ganai Katie Luo @JonasFrey96 Clark Barrett

🌐 Website: https://t.co/aEABZ3WbwO
📄 Paper: https://t.co/S34vtBELE1

[karthikpythireddi @karthikitis]

@LightningAI Just now did the DM

[karthikpythireddi @karthikitis]

Super helpful with my model training with academic subscription @lightningai https://t.co/lKiBXHPSSW

[Dhruv Diddi @DhruvDiddi]

If you're looking for a way to kick off your NVIDIA GTC weekend, join us at the Funding the Commons Hackathon at Frontier Tower:

https://t.co/6TuKemakAM

For the first time ever, all 16 floors of Frontier Tower will be activated, bringing together builders, artists, technologists, policymakers, and researchers exploring humanity’s role in the age of AI.

🚀 Solo Tech is leading the Physical AI track, bringing the largest collection of robots and inference hardware available to hack. Builders will be able to experiment with real-world systems deploying models, testing robots, and pushing embodied AI beyond simulation.

[Marco Pavone @drmapavone]

🚀 Strengthening Robot Safety with Multimodal Defenses

I’m excited to share our recent work, “Preventing Robotic Jailbreaking via Multimodal Domain Adaptation,” now available on arXiv: https://t.co/f0eJZVAAIr

As vision-language models (VLMs) become foundational components of modern robot autonomy, VLM-enabled robots also become increasingly vulnerable to jailbreaking attacks—adversarial prompts that can bypass safety filters and trigger unsafe or harmful behaviors in real-world robotic systems. This poses a significant challenge for the safe deployment of AI in autonomous vehicles, maritime robots, quadrupeds, and other embodied platforms.

📌 In this work, we introduce J-DAPT, a lightweight framework for robust multimodal jailbreak detection that delivers near-perfect detection performance across multiple robotic domains with minimal overhead.

Our results demonstrate that it is indeed possible to effectively enhance safety defenses for vision-language models in robotics—an important step toward trustworthy and reliable autonomous systems.

📄 Read the full paper: https://t.co/f0eJZVAAIr

A great collaboration with the research groups of George Pappas and Mauro Conti.

#Robotics #AI #Safety #MachineLearning #MultimodalAI

[karthikpythireddi @karthikitis]

#NeurIPS2025 workshops made all the difference for me. Especially the workshops on the VLA, world models and the Embodied AI were really amazing @nvidia @wayve_ai @physical_int

[Chelsea Finn @chelseabfinn]

I’m giving a talk at NeurIPS tomorrow morning on world models and RL for policy evaluation & improvement 🤖

Sunday at 9 am, Ballroom 20D, LAW workshop (https://t.co/4ZfMF9TUV4)

[karthikpythireddi @karthikitis]

@dpaleka Try EWM, You’ll like it

[karthikpythireddi @karthikitis]

@chelseabfinn @yoonholeee Excited for the talks.