Jimmy YU

Robotics PhD @ MIT，Undergrad @UMich

[Ao Qu @ao_qu18465]

(1/8) 💡 To make better discoveries, it can be enough to train only the idea generator.

Guidance-TTT trains a small 8B model at test time to propose better ideas, while a frozen model turns them into code. The resulting programs are then evaluated to provide the reward for updating the small model. 🔄

Using only open-weight models and no web access, we outperform the best reported results on 4 discovery tasks. 🧵

Paper and code are out:

📄 Paper: https://t.co/zHFvrOXhpw

💻 Code: https://t.co/jQPbPRmovn

[kaichen_zhou @alex_kai2020]

🚀 Honeycomb: constant-size scene memory for video world models
🐝 Six fixed-size planes with recurrent read/write.

🔁 Better revisit consistency
💾 Constant storage
⚡ ~13 ms updates

📃 https://t.co/XaXY3MQIZw
@Harvard @NUSingapore @MIT

#AI #ComputerVision #VideoGen

[Haoxiang You @haoxiang_you]

How far can coding agents go in the physical world — and can what they discover be distilled into fast System 1 robot policies like VLAs?

Introducing EmbodiedSWE, where we systematically study frontier agents across a wide range of precise, dexterous, and long-horizon robot tasks. The results surprised us: these agents could already autonomously solve remarkably complex physical tasks.

But solving is only the beginning. Running a coding agent every time a robot acts is expensive, slow, and difficult to deploy safely in the real world.

So we asked: can we turn what coding agents discover into fast robot policies?

We explore one direction: using coding-agent solutions as teachers for VLAs. A single successful solution can be expanded into a large and diverse training dataset — at surprisingly low cost.

This suggests a compelling path toward general-purpose robots: a GPT-like “brain” for reasoning and discovering solutions, paired with a fast VLA or RL “cerebellum” for acting in the physical world.

For the first time, everyday general-purpose robots don’t feel quite so far away.

[Ken Liu @kenziyuliu]

Can we build a blind, *unlinkable inference* layer where ChatGPT/Claude/Gemini can't tell which call came from which users, like a “VPN for AI inference”?

Yes! Blog post below + we built it into open source infra/chat app and served >15k prompts at Stanford so far. How it helps with AI user privacy:

# The AI user privacy problem

If you ask AI to analyze your ChatGPT history today, it’s surprisingly easy to infer your demographics, health, immigration status, and political beliefs. Every prompt we send accumulates into an (identity-linked) profile that the AI lab controls completely and indefinitely. At a minimum this is a goldmine for ads (as we know now). A bigger issue is the concentration of power: AI labs can easily become (or asked to become) a Cambridge Analytica, whistleblow your immigration status, or work with health insurance to adjust your premium if they so choose.

This is a uniquely worse problem than search engines because your average query is now more revealing (not just keywords), interactive, and intelligence is now cheap. Despite this, most of us still want these remote models; they’re just too good and convenient! (this is aka the "privacy paradox".)

# Unlinkable inference as a user privacy architecture

The idea of unlinkable inference is to add privacy while preserving access to the remote models controlled by someone else. A “privacy wrapper” or “VPN for AI inference”, so to speak.

Concretely, it’s a blind inference middle layer that:
(1) consists of decentralized proxies that anyone can operate;
(2) blindly authenticates requests (via blind signatures / RFC9474,9578) so requests are provably sandboxed from each other and from user identity;
(3) relays prompts over randomly chosen proxies that don’t see or log traffic (via client-side ephemeral keys or hosting in TEEs); and
(4) the provider simply sees a mixed pool of anonymous prompts from the proxies. No state, pseudonyms, or linkable metadata.

If you squint, an unlinkable inference layer is essentially a vendor for per-request, anonymous, ephemeral AI access credentials (for users or agents alike). It partitions your context so that user tracking is drastically harder.

Obviously, unlinkability isn’t a silver bullet: the prompt itself still goes to the remote model and can leak privacy (so don't use our chat app for a therapy session!). It aims to combat *longitudinal tracking* as a major threat to user privacy, and its statistical power increases quickly by mixing more users and requests.

Unlinkability can be applied at any granularity. For an AI chat app, you can unlinkably request a fresh ephemeral key for every session so tracking is virtually impossible.

# The Open Anonymity Project

We started this project with the belief that intelligence should be a truly public utility. Like water and electricity, providers should be compensated by usage, not who you are or what you do with it. We think unlinkable inference is a first step towards this “intelligence neutrality”.

# Try it out! It’s quite practical

- Chat app “oa-chat”: https://t.co/ELf8LvxFzX
(<20 seconds to get going)

- Blog post that should be a fun read: https://t.co/OwFmyFlZH5

- Project page: https://t.co/Swerz1xDE2

- GitHub: https://t.co/38CeKajCy2

[Jimmy YU @JimmyYu630]

Current methods may still yield geometrically valid but physically implausible reconstruction. How to fix it? (Rejection) sampling with hard physics constraints.

[Luca Carlone @lucacarlone1]

🚨Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling🚨

Foundation models for object reconstruction (e.g., SAM3D) still produce incorrect/physically implausible reconstructions. We propose PICASSO: a sampling-based approach to fix that!

[Luca Carlone @lucacarlone1]

DAAAM!!

"Describe Anything Anywhere at Any Moment".

State of the art approach to provide spatio-temporal memory to robots and agents. Powered by VLMs and scene graphs. Directly suitable for LLM queries. great work by Nicolas Gorlo and Lukas Schmid!

https://t.co/m3ISxmqgJO

[Luca Carlone @lucacarlone1]

We have completed the SLAM Handbook "From Localization and Mapping to Spatial Intelligence" and released it online: https://t.co/AnKa398nyw . The handbook will be published by Cambridge University Press. [1/n]

[Maani Ghaffari @GhaffariMaani]

A deterministic multimodal counterpart of particle filters. A new class of state space model in moment representation for optimal estimation with arbitrary noise density. Accepted for @ro

[Maani Ghaffari @GhaffariMaani]

I graduated 6 PhD students since last year!
Ray Zhang -> Research Scientist at @ToyotaResearch
@JustinTzuYuan -> Postdoc at MIT w/ @SangbaeMIT
@CynthiaChienErh -> Applied Scientist at Amazon Robotics
Tribhi Kathuria -> Lecturer at @UMRobotics
@SangliTeng -> Postdoc at UC Berkeley w/ Koushil Sreenath
Joey Wilson -> Research Scientist at @nuro
I'll try to post more about their contributions in the coming weeks.

[Andrew Davison @AjdDavison]

Thanks for the invitation to visit MIT @lucacarlone1 and great to catch up with @jleonardmit and others. John reminded me that the last time I visited was 2002! I also got to go to Boston Dynamics thanks to @czarnowskij and @tlaidlow, and the RAI Institute to see @IvanKapelyukh.

[Zhongyu Li @ZhongyuLi4]

Introducing HiLMa-Res: a hierarchical RL framework for quadrupeds to tackle loco-manipulation tasks with sustained mobility! Designed for general learning tasks (vision-based, state-based, real-world data, etc), the robot now can step over stones🐾/navigate boxes📦/dribble⚽.

[Luca Carlone @lucacarlone1]

look what just dropped right before IROS!

[Luca Carlone @lucacarlone1]

I'm enthused to see our Clio paper featured on the MIT News homepage: https://t.co/VggrpL8J7R (full article here: https://t.co/4a9GrgoTfo) [1/n]

[Heng Yang @hankyang94]

"Fast and Certifiable Trajectory Optimization" just got accepted to the International Workshop on the Algorithmic Foundations of Robotics (WAFR) @wafr_conf, great job @ShuchengK!

[Luca Carlone @lucacarlone1]

#Khronos won the Outstanding Systems Paper Award at #RSS2024 !! very proud of Lukas, Marcus, and Yun!

Khronos' code and paper are online:
Video: https://t.co/cuM2bEkSeP
Code: https://t.co/o1cRUcDp0T
Paper: https://t.co/WHwCsU06Sr

#mitSparkLab #4DSceneUnderstanding

[Heng Yang @hankyang94]

At #RSS2024?
Check out three works from the #HarvardComputationalRobotics lab presented at the main conference and workshops:

- CLOSURE: GPU-accelerated fast end-to-end uncertainty quantification for 6D pose estimation, main conference "Perception" session

- STROM: Fast and certifiable trajectory optimization with semidefinite programming running in GPUs, presented at the Frontiers of Optimization workshop and the Safe Autonomy workshop

- TRAC: Parameter-free optimizer for lifelong reinforcement learning, presented at the Lifelong robot learning workshop

[Jimmy YU @JimmyYu630]

This is super

[Heng Yang @hankyang94]

Cannot be more excited about my PhD student @ShuchengK's new work after spending his first year diving into moment and sums of squares relaxations.

Fast and Certifiable Trajectory Optimization

We have seen so many pendulum swing-ups, but I bet you haven't seen this one:

- The nonconvex trajectory optimization is solved to certifiable global optimality by a semidefinite programming (SDP) relaxation with 49,500 variables and 47,351 constraints

- How fast do we solve it? On average 0.66 seconds!

- How did we do it? A fast sGS-ADMM solver directly implemented in CUDA and runs in GPUs + data-driven warmstarts.

The method generalizes to more complicated dynamical systems:
- Cart-pole
- Car back-in (with obstacle avoidance)
- Vehicle landing
- Flying robot

Solving large-scale SDPs far beyond the reach of existing solvers.

Check out project STROM:
https://t.co/RKPAuQj6FY

[Maani Ghaffari @GhaffariMaani]

@pmddomingos It turns out that we can use a single trajectory to learn the rigid body dynamics with exactly the same performance on training and test sets. A correct by construction deep network! So you're right, and exciting to see how far we can go https://t.co/LDulpv0Yr7

[Tianyi Zhang @tianyiz2022]

[DarkGS] Building 3D Gaussians with a torch🔦 in the dark🌃
paper: https://t.co/okH7JeCQft
code: https://t.co/zMOXPPgvyj

[Heng Yang @hankyang94]

Our Harvard Computational Robotics group has
- 2 papers accepted to #L4DC2024
- 1 paper accepted to #RAL
- 1 paper accepted to #ICML2024
- 1 paper accepted to #RSS2024

80% of these paper were authored by undergrad visitors last summer, who will go to the best PhD programs this Fall (Harvard, MIT, Princeton, Stanford etc.)!

I cannot be more fortunate to have worked with them!

Their commitment, rigor, wisdom are all beyond their age.

https://t.co/dnxGeGHG5c