yisha

building robots - one at a time. incoming assistant prof @nyuniversity

[yisha @yswhynot]

If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale!
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.

[yisha @yswhynot]

If you are interested in the commercial sensors, here are the detailed links!
FSR: https://t.co/u9wBwLuihA or similar
Contact mic: https://t.co/cIymxTOKVr, sound card: https://t.co/LWd1WzpWZl (similar ones should work too)
eflesh & flexitac: details in paper, egain: customized

[yisha @yswhynot]

Touch has become a key ingredient for dexterous robot manipulation, but which tactile sensor should you actually use?

We're excited to introduce TacO, a benchmark for evaluating tactile sensors across real-world manipulation tasks.

🌮 TacO Benchmark provides:

Cross-modality comparison of different sensors (vision-, magnetic-, acoustic-, and resistive-based tactile sensing)
Standardized imitation learning framework with all kinds of tactile sensors
Open-source code, data, and hardware

Our experiments show that there is no universally best tactile sensor😌(even the expensive ones). The right choice depends on the tasks. We hope TacO helps the community build and evaluate the next generation of tactile sensors and how to use them to improve vision-based robot policies.

[yisha @yswhynot]

We also thank again @binghao_huang and @venkyp2000 for donating sensors 😆, and our friends at SNU for making and shipping eGain all the way across the ocean. We won't be able to play with so many sensors without their help!

[yisha @yswhynot]

This is led by @anyazorin , collaborating with Zilin Si, Myungsun Park, @jnsungp , @ABuynitsky , @bhadang_sachin , Taejun Park, Sohee John Yoon, Yong-Lae Park, Oliver Kroemer, Zeynep Temel, Michael T. Tolley, @xiaolonw

[yisha @yswhynot]

Project page: https://t.co/4EKaHgvhO0
Code: https://t.co/cs0qkBpXr8
Hardware: https://t.co/gJHRICSw6g
Paper: https://t.co/VPnVovv8EP

[yisha @yswhynot]

One of the best PhD defenses I’ve ever seen 🥺 I’m so jealous whoever that gets to work with Huy 🥺

[Huy Ha @haqhuy]

Last week, I defended my PhD dissertation, "Robots From Anyone: Shaping Behaviors and Embodiments with Handheld Grippers". You can find the talk here!

https://t.co/mTxhvBY4Lo

[yisha @yswhynot]

@jadenvclark @ChanghaoWang78 @YihuaiGao @Hojung_Choi_ @YifanHou2 @SongShuran Interesting work! I’m curious where exactly is the sensor placed? And does it change any physical properties of the finray effect like deformation etc? Thank you 😊

[Darrien McKenzie @darrienmckenzie]

RL training for LLMs involves exposure to problems in the “Goldilocks zone” of difficulty: not too hard, not too easy.

But difficulty is not the only thing that matters.

Problem type matters, too.

And LLMs do not see or organize problem types the way humans do.

This is the starting point of Manifold Bandits.

The fundamental issue is this: when using RL to train LLMs, there will always be problems that are more productive for the policy at a given training iteration. Across a dataset, there are an astronomical number of possible batch orderings; uniform sampling is unlikely to consistently land on the most useful ones by chance.

This is why many adaptive curriculum learning and difficulty-aware methods focus on the policy’s “optimal learning zone,” avoiding examples that are already trivial or still impossible for the LLM to solve.

But problem type is usually treated differently.

Task decompositions within a training set are often manually defined according to human semantics, or ignored entirely, with an entire dataset/domain treated as “one task,” despite the immense heterogeneity seen even in small training datasets.

It is generally understood that LLMs may not share our concept of difficulty. But it seems less recognized that LLMs may not share our concept of problem type, either. In both cases, training the LLM according to our own semantics can obscure fine-grained learning dynamics that could otherwise be exploited.

In this work, we seek to derive an adaptive curriculum learning method that caters not only to the policy’s ability, but also its perception: its latent organization of tasks.

Driven by the manifold hypothesis and the Bitter Lesson, we leverage computation to derive curriculum structure from the policy’s latent geometry, then use Bayesian inference over that structure to guide search and learning over the training set. The result is Bayesian Manifold Curriculum (BMC): an algorithm that does not just try to find problems of the right difficulty, but instead orchestrates training effort across diverse and interacting problem types.

More technically, we frame problem sampling for LLMs as a manifold-structured bandit problem with endogenous non-stationarity. The tl;dr is that the problems/arms are diverse, structured, and interact through policy updates, so the typical non-stationary bandit framing is not quite the right fit for training LLMs. BMC is derived from this new framing.

🌐Website: https://t.co/iKVjNK92up
📰Paper: https://t.co/j08GktD6Qv
💻Code: https://t.co/ki6O3ojMWO

🧵1/N

[Binghao Huang @binghao_huang]

Train on one tactile and deploy on another

[yisha @yswhynot]

We show that the policy trained with one tactile sensor can be transferred to another sensor! Also, great thanks to @binghao_huang and @venkyp2000 for donating sensors :D

[Sachin Bhadang @bhadang_sachin]

A robot policy trained on one tactile sensor usually breaks the moment you swap the sensor. New hardware → recollect data → retrain.

We built TACTX: train tactile policies on one sensor, deploy zero-shot on a physically different one.

The catch, these sensors don't just look different, they measure contact through totally different physics:
👁️ Daimon — vision-based · 🧲 eFlesh — magnetic · ⚡ FlexiTac — resistive
Same contact but completely different raw signals. 🧵

[yisha @yswhynot]

This so fun to play with!

[Nicklas Hansen @ncklashansen]

Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨

We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict *when* hallucination happens and use that to fix it!

🧵1/n

[yisha @yswhynot]

@iamRezaSayar It’s the Dynamixel ones

[yisha @yswhynot]

If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale!
We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.

[yisha @yswhynot]

@qiayuanliao Thank you for posting!!

[yisha @yswhynot]

@anna_macalus Thank you!!🥰

[yisha @yswhynot]

@woodtechwill This also works on synthetic trajectories too :)

[yisha @yswhynot]

@BaoyuLi6 Good question🙌 One reason is we noticed many human demos are dominate by these two fingers. Another more realistic reason is more fingers lead to a lot more collisions and hard to manufacture, so we just start with an easier problem :)

[yisha @yswhynot]

We also find that by simply making the hardware optimizable with control policy, the fingertip tracking error can go down to less than 1mm - not possible for the commercial robot hands we tried of the same dof

[yisha @yswhynot]

This was quite a weird project when I started a year ago🦭 but the designs that came out of 3d printers are so fun to play with!

[yisha @yswhynot]

with my awesome co-authors @ncklashansen @baicrystal25 @carlo_sferrazza Mike Tolley @xiaolonw

[yisha @yswhynot]

Paper: https://t.co/MPbQaWZGGW
Webpage: https://t.co/aggqhuTSVS