김민수

[Seohong Park @seohong_park]

Behavioral cloning mystery

https://t.co/VqxzvcfGSx

I wrote a new blog post about "mysteries" in behavioral cloning that appear with real-world robot data (e.g., overfitting is "good"). I also tried to demystify them and shared my thoughts!

[Eren Chen @ErenChenAI]

Why Robots work in Simulation but fail in Reality

One of the most frustrating moments in robotics:

Everything works perfectly in simulation. Then you deploy it on a real robot and suddenly:

The grasp misses
The arm shakes
The robot drifts
Contact becomes unstable
The motion looks correct, but the task still fails

How do you solve the sim to real problem?

At first, it sounds simple. Just move the code from simulation onto hardware.

But the gap between simulation and reality is much larger than most people think.

Simulation environments are extremely clean.

The table is flat.
Object geometry is accurate.
Friction is predefined.
Sensors are stable.
Robot joints behave exactly as expected.

But the real world is messy.

Lighting changes.
Depth sensors drift.
Objects reflect light differently.
Motors have delay.
Joints have backlash.
Contact forces behave unpredictably.
And robotics is a chain reaction.
A small perception error becomes a planning error.
The planning error becomes a control error.
The control error becomes an execution error.

Eventually, the robot misses the grasp by a few centimeters and the entire task fails.

And The hardest part is usually contact.

Humans think tasks like: grasping a cup, opening a door, inserting an object, pushing a box are trivial.

For robots, these are extremely difficult because contact is not clean physics. A tiny shift in friction, force, or surface geometry can completely change the result.

In simulation, objects are usually “well behaved.”
In reality:
objects slip
contact points shift
surfaces deform
collisions happen unexpectedly

This is why many robotic tasks fail not because the policy is fundamentally wrong, but because reality itself introduces uncertainty.

Sensors are also less reliable than people think.

The robot’s perception already contains error:
camera noise
unstable depth estimation
occlusion
pose estimation drift
changing lighting conditions

Sometimes the model itself is fine, but the input is already slightly wrong. By the time the error propagates to the end effector, the grasp fails.

The robot hardware itself is also imperfect.
Motors have latency.
Controllers have frequency limits.
Actuators have error.
Different loads change behavior.

In simulation, the robot follows commands perfectly.

In reality, it may move slightly slower, slightly off target, or slightly unstable. Those tiny differences are fatal in robotics because robots physically interact with the world.

Sim2Real being difficult does not mean simulation is useless. Simulation is still incredibly valuable: they are cheap, safe, scalable and reproducible.

A better way to think about simulation is: Simulation is the training ground, not the final battlefield.

Modern Sim2Real methods usually combine multiple approaches: making simulation more realistic, adding domain randomization, randomizing lighting, friction, object positions, and sensor noise, fine-tuning with real-world data.

The goal is not to make the robot adapt to one perfect virtual world. The goal is to make the robot robust enough to survive an imperfect real one.

The most important lesson in robotics is:

Success in simulation is only the first step. The real test begins when the robot touches the real world.

Video Credit: Kevin Zakka

[Weishuai Zeng @weishuaizeng]

We are excited to re-introduce our Behavior Foundation Model for Humanoid Robots, built upon a unified perspective of diverse WBC tasks, as a promising step toward a foundation model for general humanoid control.
🔗Website: https://t.co/AMDocTClqD
📜Paper: https://t.co/ema9egIJcE

[김민수 @_hhebb]

📑 Preprint (arXiv): https://t.co/q7LrjtFDHO 🎥 Video: https://t.co/kmoyG9GH7Q thanks to all our collaborators—couldn’t have done this without your dedication! Hope to see you in Vancouver, Canada!

[김민수 @_hhebb]

🎮⚽ Do you like football or football games? ⚽🎮

🎉 I’m excited to share that our paper has been accepted to ACM SIGGRAPH 2025, journal track! 🎉
🔹 PhysicsFC: Learning User-Controlled Skills for a Physics-Based Football Player Controller. Minsu Kim, Eunho Jung, Yoonsang Lee*

[김민수 @_hhebb]

🔹Topic: A framework for controlling physically simulated football players to perform dribbling, trapping, moving, and kicking—seamlessly blending realism and interactivity.

[Google DeepMind @GoogleDeepMind]

We’re presenting AlphaProteo: an AI system for designing novel proteins that bind more successfully to target molecules. 🧬

It could help scientists better understand how biological systems function, save time in research, advance drug design and more. 🧵 https://t.co/lx35RvplFr

[김민수 @_hhebb]

I just had 150 Rainbow Points dropped into my wallet — everybody has at least 100 points waiting for them, but you might have more! Claim your drop: https://t.co/BIMouu2Px0

[Elon Musk @elonmusk]

Starship launch on track for tomorrow morning

[Elon Musk @elonmusk]

Preparing for launch on Saturday

[AK @_akhaliq]

Meta presents EMU VIDEO: Factorizing Text-to-Video Generation by Explicit Image Conditioning

blog: https://t.co/tjcX2M0Ear
paper: https://t.co/awIgv5Eyrq

present EMU VIDEO, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions–adjusted noise schedules for diffusion, and multi-stage training–that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work– 81% vs. Google’s Imagen Video, 90% vs. Nvidia’s PYOCO, and 96% vs. Meta’s Make-A-Video. Our model outperforms commercial solutions such as RunwayML’s Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user’s text prompt, where our generations are preferred 96% over prior work.

[AI at Meta @AIatMeta]

New AI research paper from Meta — MART, or Multi-round Automatic Red-Teaming is a framework for improving LLM safety that trains an adversarial and target LLM through automatic iterative adversarial red-teaming.

Details in the paper ➡️ https://t.co/lGuJrPDsCD

[AK @_akhaliq]

Drivable 3D Gaussian Avatars

paper page: https://t.co/GfUShcV1ZZ

present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

[Greg Brockman @gdb]

After learning today’s news, this is the message I sent to the OpenAI team:

[Sam Altman @sama]

i loved my time at openai. it was transformative for me personally, and hopefully the world a little bit. most of all i loved working with such talented people.

will have more to say about what’s next later.

🫡

[Pika @pika_labs]

Pika 1.0, Coming Soon.

Check out our 3D animation teaser made completely by text-to-video AI. 🙌

[Sergey Levine @svlevine]

Check out @Andre3035858461 & @vivek_myers's blog post on GRIF, our method for vision-language policies with language alignment: https://t.co/FfU32K35cY

Semi-supervised learning of language following policies.

[Yilun Du @du_yilun]

Introducing Video Language Planning!

https://t.co/N67pVk6BJh

By planning across the space of generated videos/language, we can synthesize long-horizon video plans and solve much longer horizon tasks than existing baseline (such as RT-2 and PALM-E).

(1/5)

[Mitsuhiko Nakamoto @mitsuhiko_nm]

How can we leverage Stable Diffusion’s internet-scale pre-trained knowledge to enhance zero-shot robotic manipulation?

Check out our new paper😉

SuSIE: Subgoal Synthesis via Image Editing
https://t.co/VW2X9EaK0x

Detailed threads from Sergey and Aviral below:

[Sergey Levine @svlevine]

Neural nets extrapolate unpredictably, right? It turns out that for reasonable shifts, extrapolation can be much more structured: neural nets can be made to "hedge" for OOD inputs, reverting to the "optimal constant prediction" (OCS). A thread about our recent paper below: