xun

Software engineer by trade. Ex-Google. Ex-Pinterest. Tinkering with so100.

[xun @paszea]

The ground is shifting for robotics.🤖 And @LeRobotHF made it easy for all to experience this shift firsthand.✨

I wrote my experience here https://t.co/gyiuEt7SBz (w/ videos, datasets, learnings!) The gist:  a total time-suck but the fun & learning 💯 worth it.

[xun @paszea]

@OfirOzeri You can just add a gripper position to the IK output and call the standard send_action() of the robot class. This way you have flexibility to do things like: open the gripper at 50 degrees, move it to a point, close the gripper, move it to a drop point, open it, etc.

[xun @paszea]

So100 costs $110. I has no fancy features. You control it by setting its 6-joint positions.

But you can develop a fancy feature, like a simplified IK to move the arm to an arbitrary position in images. See

[xun @paszea]

@OfirOzeri Thanks! 

Once you have a point in the image (e.g. point of an object.) IK can map it to an action. To smooth the move, I linear-interpolate some intermediate actions btw the arm's current state and the action. Send these actions to the arm with some sleep in between.

[xun @paszea]

@leo_lin6 Sounds like a perfect use case for blind ACT :)

[xun @paszea]

Blind ACT - a poor man's IK

How do you like an ACT model that grasps like a pro, but is 4x as fast and invariant to scene changes?

The so100 in the demo is controlled by a policy that sees nothing. The image on the laptop is for humans to visualize. The robot acts blind. 1/n

[xun @paszea]

@OfirOzeri Gemma models might not be powerful enough. But it doesn't hurt to give it a try. Suggest to get it to work with Gemini 2.5 first to rule out other factors.

[xun @paszea]

In-context learning to control so100

Give gemini-2.5-flash a few examples. It learned to control my so100:

[xun @paszea]

Fascinating! 🤯 Powerful pre-trained LLMs are easier to RL, and the same turns out true for VLAs. The trick? Training a lightweight noise sampling policy, completely sidestepping expensive diffusion model backprop. So clever!

[Sergey Levine @svlevine]

If you have a policy that uses diffusion/flow (e.g. diffusion VLA), you can run RL where the actor chooses the noise, which is then denoised by the policy to produce an action. This method, which we call diffusion steering (DSRL), leads to a remarkably efficient RL method! 🧵👇

[xun @paszea]

@xiao_ted Big fan of Gemini's embodied reasoning! 🧠 It's proving incredibly useful in my projects and I'm consistently impressed with its capabilities. ✨

[Ted Xiao @xiao_ted]

Gemini 2.5's multimodal embodied reasoning capabilities are making a huge difference for many robotics researchers and industry developers.

The Gemini Robotics team has curated a blog post of use cases, recommended prompts, real-world examples, and partner case studies 🧵

[xun @paszea]

@pravsels It's a two step process with me in the middle to confirm a bbox is correct before asking for trajectory.

[xun @paszea]

@pravsels Also I recorded at 30fps since I didnt want to miss any important actions. I tested at various down sample rates to see which one worked the best. You might need to do the same for your case

[xun @paszea]

@pravsels Teleop style also helps. Besides what I mentioned in the post, you want to keep the style as consistent as possible

[xun @paszea]

@IliaLarchenko Yes. I used simple linear extrapolation to move the arm. To avoid collision I made it to move up before placing. It'd be interesting to ask LLM to come up with trajectories which I haven't tried.

Re cameras, your ideas should work if you could get a good fov from both.

[xun @paszea]

@WillNorris2 @LeRobotHF I didn't try bbox with Groot N1 or Pi0 since it already worked well with act. I believe @shreyasgite tried it on pi0.

[xun @paszea]

The ground is shifting for robotics.🤖 And @LeRobotHF made it easy for all to experience this shift firsthand.✨

I wrote my experience here https://t.co/gyiuEt7SBz (w/ videos, datasets, learnings!) The gist:  a total time-suck but the fun & learning 💯 worth it.

[xun @paszea]

Here is the link in case bitly doesn't work
https://t.co/BhlcOa4BLp

[xun @paszea]

All zero shots. The key enabler is the IK model I developed earlier https://t.co/a1aoOhs6Jv , and gemini live to put everything together.

[xun @paszea]

So100 costs $110. I has no fancy features. You control it by setting its 6-joint positions.

But you can develop a fancy feature, like a simplified IK to move the arm to an arbitrary position in images. See

[xun @paszea]

#zeroshot #so100
A robot that talks too much and pauses too often, yet performs tasks Zero Shot. No demonstration needed.

1/ Put things in a plate

[xun @paszea]

4/ Or hand me a dice