Revant Mahajan

Generalised Robotic Exploration in Real World | M.S. CSE @UCSD | Ex-Software Engineer in Path Planning @AmazonRobotics

[yisha @yswhynot]

I’ve been joking with friends that my main job this year has been interviewing 🫠

Now I’m finally excited to share what’s next: I’ll be joining NYU as an Assistant Professor next fall, and before that, I’ll be spending a year at @physical_int !

I feel incredibly fortunate to have spent the past two years at UCSD working on control-hardware co-design.

When I was looking for a postdoc, I told people I wanted to work on robot learning for hardware design. I’m deeply grateful that @xiaolonw and Mike trusted me enough to start something none of us quite knew how to do.

Whenever I mentioned working with both of them, people either knew only one of them (they come from very different communities), or found the combination surprising. I find it to be really fun and I learnt so much more than I would anywhere else.

Although I think jointly optimizing policies and hardware at scale is still somewhat early, I strongly believe in optimizing (or at least iterating) hardware alongside policy training.

Over the past few decades, we’ve developed a lot of depth in individual hardware components: sensors, actuators, mechanisms, materials, etc. But we still haven’t explored enough of their combinations as integrated systems, or understood their actual usefulness for policy.

As coding agents get more and more powerful, learning where and how hardware is bottlenecking robots is really valuable. And the only way to understand this is to scale up and try things :)

I’m getting ready to join the bay area robotics community, but it’s still a bit sad to leave the beautiful San Diego.

I had a lot of fun sampling taco shops on Tuesdays (although I eventually converged to two places on my way home...). I will definitely miss the on campus climbing gym, and the forever-sunny-and-warm outdoor swimming pool next to it 🥺

[Revant Mahajan @MahajanRevant]

Conveyors have their own sets of challenges

1. It’s a fixed investment in terms of structure. A specific layout now is not necessary the best layout for future where you have new technologies you can integrate. One of the biggest headaches when it comes to brownfield to greenfield migration for solutions

2. They get jammed a lot! The belts have their own annoyance!

3. This design is a very good idea because of the modularity. Allows easier change of design based on actual performance of the system. Although i suspect the cost is somewhat higher?

[Revant Mahajan @MahajanRevant]

Is this me or is Opus 5 a regression over opus 4.8 with respect to intent understanding and asking clarifying questions?

[Revant Mahajan @MahajanRevant]

This is so damn creative and cool I love it! I absolutely love this! We need more mini robots working with bigger robots hahaha!

[Mingyu Ding @dingmyu]

🤖 A robot that transforms between humanoid and dexterous hand?

Introducing Handroid, a reconfigurable robot with a shared 27-DoF body 🖐️→🚶→🖐️:
• The same joints, different roles
• Expanded robot task space
• Shared sensing and control interfaces

https://t.co/wCVGuo9MmJ

[Revant Mahajan @MahajanRevant]

@Stone_Tao @physical_int Congratulations Stone!!

[Revant Mahajan @MahajanRevant]

@patience_cave This is so cool and interesting
Might be my new binge game hahaha 😂

[Paul Zhou @zhiyuan_zhou_]

This is a robot failing to grasp a ball. Almost every robot lab produces clips like this daily… and almost all of them get thrown away. This is the most abundant but underused resource in robot learning. We’re collecting all of it now as ✨OopsieData✨, please join us!

[Revant Mahajan @MahajanRevant]

@agupta I am actually curious how they solve the problem of alignment within their agent harness? How would you stop the model from accidentally knocking over something? Although I’ll admit this seems more tractable if you use agents than if you relied purely on a policy

[yisha @yswhynot]

Touch has become a key ingredient for dexterous robot manipulation, but which tactile sensor should you actually use?

We're excited to introduce TacO, a benchmark for evaluating tactile sensors across real-world manipulation tasks.

🌮 TacO Benchmark provides:

Cross-modality comparison of different sensors (vision-, magnetic-, acoustic-, and resistive-based tactile sensing)
Standardized imitation learning framework with all kinds of tactile sensors
Open-source code, data, and hardware

Our experiments show that there is no universally best tactile sensor😌(even the expensive ones). The right choice depends on the tasks. We hope TacO helps the community build and evaluate the next generation of tactile sensors and how to use them to improve vision-based robot policies.

[Nicklas Hansen @ncklashansen]

Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨

We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict *when* hallucination happens and use that to fix it!

🧵1/n

[Revant Mahajan @MahajanRevant]

I agree with the special traits! Although there is a point to be made how bad we are expressing non trivial goals in language. Compounds significantly worse for physical task where it’s very verbose to express certain way you want things done or certin things to avoid. Often easier to geometrically reason sometimes and communicate

If we were able to explain everything in language perfectly it would be the only thing. Sadly (or not sadly) we other modalities to help us cleanly articulate our intentions

[yisha @yswhynot]

If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale!
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.

[Hyunjin Kim @hjhyunjinkim]

What if a robot could learn the physics of a soft object, how a towel folds or lift a plush toy, by watching you play with it from a first-person view?

Introducing EgoPhys. We build deformable twins from one egocentric RGB video using a generalizable material codebook.
[1/6] 🧵

[Revant Mahajan @MahajanRevant]

@agupta New England fall is absolute beauty! Although I’ll say San Diego’s kinda better than both haha 😛

[Revant Mahajan @MahajanRevant]

@KyleVedder @JunyaoShi Thanks! Will do :)

[Revant Mahajan @MahajanRevant]

@KyleVedder @JunyaoShi In recent times, any specific academic paper you read where you thought they got it right?

Trying to understand a bit more in depth what you mean. Figured reading such a paper would give me the best intuition

[Revant Mahajan @MahajanRevant]

@KyleVedder @JunyaoShi What kind of regimes do you think people in academia should focus on?

[Revant Mahajan @MahajanRevant]

@_arohan_ The bear was very happy to see everyone graduate haha 😂

[Revant Mahajan @MahajanRevant]

@litian_liang Hahahahah that’s sick!!!

[Revant Mahajan @MahajanRevant]

@tankots @WisprFlow Not a hater and a consistent user

The floating thing can be annoying at times and activates when I am trying to work with the Mac dock. Maybe it can use the menu bar widgets if the wispr as is?