Gagan Khandate

Staff Research Scientist @BostonDynamics, PhD @ColumbiaCompSci, undergrad from @iitmadras

[Jitendra MALIK @JitendraMalikCV]

At the RI seminar at CMU yesterday, I presented a 3 level analysis of robot skills & discussed the pros and cons of teleoperation, simulation, and learning from videos, before presenting our research. Enjoy! https://t.co/wUPe3QlTRk

[60 Minutes @60Minutes]

60 Minutes gets a look at the first real-world test of Boston Dynamics’ humanoid robot Atlas, offering a glimpse of a future coming faster than you might think. Sunday. https://t.co/mEN4CWeXMW

[Siqiao Huang @KnightNemo_]

Introducing🌍 Awesome-World-Models, a one-stop github repo of everything there is to know about world models!
Here is a new, curated one-stop resource list for everyone interested in "World Models," aiming to be a go-to guide for researchers and developers in the field.
🧵(1/n)

[Gabe Margolis @gabe_mrgl]

Excited to share SoftMimic -- a new approach for learning compliant humanoid policies that interact gently with the world.

[Alejandro Escontrela @alescontrela]

Simulation drives robotics progress, but how do we close the reality gap?

Introducing GaussGym: an open-source framework for learning locomotion from pixels with ultra-fast parallelized photorealistic rendering across >4,000 iPhone, GrandTour, ARKit, and Veo scenes!

Thread 🧵

[Sergey Levine @svlevine]

I always found it puzzling how language models learn so much from next-token prediction, while video models learn so little from next frame prediction. Maybe it's because LLMs are actually brain scanners in disguise. Idle musings in my new blog post: https://t.co/BxInvWx4y0

[Seohong Park @seohong_park]

Is RL really scalable like other objectives?

We found that just scaling up data and compute is *not* enough to enable RL to solve complex tasks. The culprit is the horizon.

Paper: https://t.co/KsNZgk782S

Thread ↓

[Shuran Song @SongShuran]

What makes a robot hand design better at learning from human demonstrations?

Is it being similar in size to a human hand, or matching its degrees of freedom?

DexMachina lets us explore this question in simulation — and the results are quite interesting! Check it out 😉

[Sergey Levine @svlevine]

We came up with a really simple way to train flow matching (diffusion) policies with offline RL! Flow Q-learning from @seohong_park uses a distillation (reflow-like) scheme to train flow matching actor, and works super well!

Check it out: https://t.co/TYYXGuyAgI

[Shuran Song @SongShuran]

🚀 Meet ToddlerBot 🤖– the adorable, low-cost, open-source humanoid anyone can build, use, and repair!

We’re making everything open-source & hope to see more Toddys out there!

[David D. Baek @dbaek__]

1/9 🚨 New Paper Alert: Cross-Entropy Loss is NOT What You Need! 🚨
We introduce harmonic loss as alternative to the standard CE loss for training neural networks and LLMs! Harmonic loss achieves 🛠️significantly better interpretability, ⚡faster convergence, and ⏳less grokking!

[Chris Paxton @chris_j_paxton]

I've been waiting for this for a while. Open source procedural scene generation from NVIDIA. This kind of thing would be really useful for scaling up simulation data for robots.

[Jim Fan @DrJimFan]

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions.

I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control.

We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics.

Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs.

The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

[Boston Dynamics @BostonDynamics]

Atlas doing a quick warm up before work.

[Autonomous Robots @AUROblog]

🔍This study focuses on overcoming the challenges of #ReinforcementLearning (RL) for motor control policies in complex tasks such as #dexterousmanipulation.
🔗Check it out: https://t.co/a1ht4xotL9
@GaganKhandate @XiaoYangLiu10 @ColumbiaCompSci @CUSEAS @Columbia

[Soroush Nasiriany @snasiriany]

Over 50 researchers in the robot learning community joining forces on a mission to scale up robot learning to an unprecedented level 🚀 It’s amazing to see what we can achieve as a team! I made so many new friends in the process and I’m truly grateful for that ❤️

[Hao Su @haosu_twitr]

Introducing ZeroRF, where we reconstruct high-quality radiance fields from *sparse views* at an order of magnitude faster than previous methods (30 secs for 320x320) without any pretraining or additional regularization!

https://t.co/9XCRqrgX0r

[Chris Paxton @chris_j_paxton]

Represent robot policies as trajectories through space. Naturally allows for cross-embodiment transfer and learning from human video!

[Yann LeCun @ylecun]

Let me clear a *huge* misunderstanding here.
The generation of mostly realistic-looking videos from prompts *does not* indicate that a system understands the physical world.
Generation is very different from causal prediction from a world model.
The space of plausible videos is very large, and a video generation system merely needs to produce *one* sample to succeed.
The space of plausible continuations of a real video is *much* smaller, and generating a representative chunk of those is a much harder task, particularly when conditioned on an action.
Furthermore, generating those continuations would be not only expensive but totally pointless.
It's much more desirable to generate *abstract representations* of those continuations that eliminate details in the scene that are irrelevant to any action we might want to take.
That is the whole point behind the JEPA (Joint Embedding Predictive Architecture), which is *not generative* and makes predictions in representation space.
Our work on VICReg, I-JEPA, V-JEPA, and the works of others show that Joint Embedding architectures produce much better representations of visual inputs than generative architectures that reconstruct pixels (such as Variational AE, Masked AE, Denoising AE, etc).
When using the learned representations as inputs to a supervised head trained on downstream tasks (without fine tuning the backbone), Joint Embedding beats generative.

See the results table from the V-JEPA blog post or paper:
https://t.co/mfLvtvk8jj

[Cheng Chi @chichengcc]

Can we collect robot data without any robots?

Introducing Universal Manipulation Interface (UMI)

An open-source $400 system from @Stanford designed to democratize robot data collection

0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)