Ishaan Shah

ML/Robotics @SkildAI

[Deepak Pathak @deepakpathak]

"Does S1 exhibit physical prompt steerability: different prompts induce distinct behavior in the same environment?"

Yes! Watch S1 follow 4 different video prompt recipes in the same kitchen.

The first one is something far out of distribution -- "putting a plate in a toaster".

[Deepak Pathak @deepakpathak]

You don't have to post-train ChatGPT on every user. If you did, it would never have taken off. Yet this is exactly how robotics works today

For robots to take off, they need to learn in-context.

Great article from @Chris_J_Paxton on this new paradigm and S1's place in it:

[Deepak Pathak @deepakpathak]

One surprising aspect of S1’s in-context learning (ICL) is where it shines most: super long-horizon tasks (10+ min) and scenarios outside the pretraining distribution.

For short, simple pick-and-place tasks (5–20 sec), most frontier models can already perform well either zero-shot with no prompt or from just a language prompt. With enough pretraining data, the model often already “knows” what needs to be done. @DrJimFan also highlighted this distinction eloquently here: https://t.co/vbZWh3NmTy

Once you move to genuinely unseen scenarios and long-horizon tasks, in-context learning becomes critical. This shows up dramatically in the scaling laws: S1’s ICL advantage over regular VLAs grows exponentially as we move further out of distribution.

We discuss this in detail in the S1 blog:
https://t.co/xpUtioZM5K

[Ishaan Shah @ishaankshah]

@JunyaoShi @JaskaranG408 Thank you @JunyaoShi !

[Ishaan Shah @ishaankshah]

Thrilled to share what I've been working on!

This figure from the blog is what makes me excited about pre-training in robotics today. With scale and diversity, performance of in-context learners like S1 predictably improves and far exceeds conventional VLAs on OOD tasks! (Robotics is far more data-bottlenecked than LLMs and most real world use-cases are OOD)

Here's one of the most interesting interactions I've had with S1. We bought a skateboard to try and see if S1 could perform wheel assembly - a precision task that wasn't seen in pre-training. Prompted with a single video demonstration consisting of wrist camera streams, S1 consistently aligns the wheel with the axle. This kind of skill would have taken a couple hours of data + fine-tuning a few months ago. Even better - when it misses the alignment - it recovers from this mistake by re-grasping and adjusting the wheel - a sign of how S1 understands intent even in OOD scenarios!

[Skild AI @SkildAI]

Introducing S1, our new foundation model that learns from one example.

It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning.

Watch S1 operate in real-time via in-context learning:

[Deepak Pathak @deepakpathak]

We’re scaling S1 as fast as we can. The curve shows no signs of slowing down, steadily going up.

We’ve been working for a while to find this kind of scaling law in robotics.

[Jiachen Zhu @JiachenAI]

Best barista at @SkildAI!

[Jiachen Zhu @JiachenAI]

Jas has been quietly cooking. Keep an eye on this guy!

[Deepak Pathak @deepakpathak]

This is a good question. If the task was only 5–10s long, it can be difficult to distinguish true physical steerability from generalization. But we’re doing tasks that run for over 10 minutes and not seen during training (OOD). Sustaining performance at such long horizon and novel tasks would be impossible without physical steerability actually happening. Hope this clarifies.

[Chris Paxton @chris_j_paxton]

Robots cooking eggs

[Abhinav Gupta @gupta_abhinav_]

Making eggs is an extremely difficult task. Even my seven year old son still struggles to break eggs perfectly.

But @SkildAI foundation model performs this task almost perfectly. The large-scale pre-training also allows some interesting recovery strategies.

[Deepak Pathak @deepakpathak]

We hosted Prof. Alyosha Efros (UC Berkeley) at @SkildAI! He didn't believe that robots could actually cook eggs reliably. :)
Tested back-to-back 5times without fail! One batch of scrambled eggs every ~2.5mins nonstop. The same model assembles a GPU on a server rack too.

[Essential AI @essential_ai]

[1/5]

We have a quick update to share, which contradicts our hypothesis regarding the abilities of Muon and Adam vis-a-vis Grokking.

[Andrew Hojel @AndrewHojel]

Check out what the Data Team at @essential_ai has been cooking! It's been a blast preparing this dataset and super excited to see what people use it for. Shoot me a DM with any questions or cool use cases.

[Essential AI @essential_ai]

Muon is a serious competitor to AdamW, but it's tricky to scale up.

Our infra team has made fundamental advancements in parallelizing Muon on large scale distributed clusters. We're extremely happy with the result and it's now a part of our pretraining pipeline.

🔗Check out our new blog post on this:
https://t.co/YwLswGIBSF
https://t.co/zrWef7pUBI

@pcmonk @YashVanjani @ishaankshah @karlstratos @ashVaswani

[Ashish Vaswani @ashVaswani]

Terrific work speeding up Muon led by @pcmonk, in collaboration with @YashVanjani , @ishaankshah , and @karlstratos . We’re exited about the benefits we’re seeing with Muon.