Eligibility (strict):
📍 Full-time, in person in Bengaluru (100% in office)
🗓️ 6 months, Jan–Jun 2027
🎓 Current students (final year preferred) and recent grads
Apply here: https://t.co/AN41Ey9aPC
Hiring an intern for my team at Microsoft Research India!
We are ambitious builders working on Human-centered AI (HCI x product x applied ML). We iterate and ship fast, and you'll get to own ambitious projects.
Details + form below🧵
Can a robot manipulate a rope, granular objects or cloth to reach precise 3D shapes?
🚨Introducing DefFlow: a goal-conditioned 3D World Action Modelling approach that learns to manipulate deformable objects into diverse goal shapes using 3D point flows.
A thread🧵(1/n)
@LinusMixson@barrowjoseph@teortaxesTex Yup, my bad!
The featureless blob is indeed the convex hull.
Shadowbench is intended more as a study of model behavior.
As such, it is weird how every model identifies the animal from the silhouette but then makes no attempt to capture the animal's anatomy via hand poses
@yacineMTB@giffmana@teortaxesTex Real 🥀
With all the 3D + robotics demos around I wanted to probe spatial understanding+dexterous manipulation in a more 'raw' sense.
@giffmana@teortaxesTex Hey Lucas!
Creator of shadowbench here, and yup these are my intended takeaways from the experiment!
1. A featureless blob outscores any attempt by even opus
2. Models seem to optimise 'blobs' instead of recreating the animal.
@janekm Thanks Janek!
@perpetual_si the people want me to burn more tokens let me burn more tokens
This is an esoteric book from 1859, highly doubt it has been trained on (the eval results seen to agree!)
Introducing ShadowBench: can frontier models recreate shadow puppets?
The model gets an RL environment with a curtain and 3D hands, and must pose them joint by joint until it matches a diagram from a 150 year old (!) puppet book.
Crafted at @perpetual_si
@andrezfu Real 🥀
It's quite crazy how literally all the models identified what animal the silhouette was... and then made no attempt whatsoever to recreate the animal using the hands.
@justALEXWORTEGA Hey Alex, cool experiment!
The point of my work was simply to examine how models would behave in this setting. My two main findings are:
1. A featureless blob outscores any attempt by even opus
2. Models seem to optimise 'blobs' instead of recreating the animal.
@pvibhat@perpetual_si Thanks Pruthvi!
More evals coming soon, do check out the dataset on HF in case you'd like to tinker around!
https://t.co/uqG68gO5y1
@ahmedmuhi01@perpetual_si Thanks Ahmed!
Disagree about this being hard for humans. Check out Henry Bursill's Hand Shadows to be Thrown upon the Wall (1859).
Quite easy to follow directions, something LLMs here were clearly not able to do haha!
@kvdozer@perpetual_si Thanks! With all the 3D + robotics demos around I wanted to probe spatial understanding+dexterous manipulation in a more 'raw' manner.
Simple IoU made sense as I lacked 3D pose ground truths for this (no such dataset exists for shadow puppets!)