Xiao-ke (XK)

Econ, robots, debate, fish 🐠

[Lucas Beyer (bl16) @giffmana]

Another cool update from the robotics world! While we're all wow-ed in mental land, there's a ton of progress and i feel like acceleration happening in physical land too now.
My guess is in good part also (but not only) thanks to the coding models progress over the last year.

[Jannik Schilling @Jannikschg]

In 8 weeks, we scaled our UMI data collection to 1M+ diverse tasks at $10 per hour of data.

How? Scripted tasks have low throughput, so we made a bet: collect massive amounts of freeform data, with operators interacting with objects on a table however they like. That gave us 5x+ more data than scripted collection.

The catch: freeform data is unusable without annotation, and VLMs couldn't annotate at scale yet when we started Pantheon.

Last month, VLMs finally got good enough. The bet paid off.

Check out the full blog post for more on our hardware, infra, and how a core team of 5 scaled to 90 operators in a few weeks!

[Xiao-ke (XK) @xklionfish]

@calixo888 @PantheonInc So excited!!! 💥

[Xiao-ke (XK) @xklionfish]

@mesomemore Appreciate you!!

[Xiao-ke (XK) @xklionfish]

@mesomemore we make robot

[Xiao-ke (XK) @xklionfish]

So excited about what we’ve built so far- much more to come! 🚀🚀

[calix @calixo888]

How do you build the most diverse UMI dataset in robotics?

In the last 8 weeks, we scaled an international UMI data collection operation from 5 to 90 operators, built out a dataset of over 1M+ unique tasks, and continue to capture over 45k+ new tasks each day.

Robotic World Models require much higher diversity of tasks than imitation learning. Here's the why and how behind our operations. 🧵

[Jannik Schilling @Jannikschg]

When we first built a VLM annotation pipeline back in April, we found that VLMs were nowhere near good enough to be used in our data pipeline.

Just six months later, VLMs are better than most human annotations we looked at, and can be run at orders of magnitude more scale, and much cheaper.

And this is the worst the models will ever be! We are very excited for all the opportunities that this opens up for research using large-scale data.

We open sourced the pipeline, and you can try it out on our website:

[Xiao-ke (XK) @xklionfish]

@calixo888 @PantheonInc @ericli_ 🚀🚀🚀

[Xiao-ke (XK) @xklionfish]

Super excited about Argus and the potential it has to improve the data we train on. Try it here: https://t.co/IjioYRiYFi

[calix @calixo888]

Argus: an open-source robotics data annotation and quality pipeline.

With new frontier VLMs like GPT-6 Astra, we can generate rich, high-quality annotations for robotics data for both training and dataset analysis. Argus delivers detailed, timestamp-level annotations while catching issues like mislabeled instructions, sped-up recordings, swapped camera streams, and unflagged operator mistakes.

To support better data quality for the robotics community, we’re open-sourcing Argus for anyone to use 🧵

[Dhananjay Ashok @Dhananjay99Ash]

Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games:

Paper: https://t.co/05Fpom1OM1
Website: https://t.co/qFUZfld80b

[Xiao-ke (XK) @xklionfish]

@Dhananjay99Ash Super cool, congrats @Dhananjay99Ash!

[Jannik Schilling @Jannikschg]

When we trained robotics models, we were wondering why certain datasets produced way better results than others.

Over the last two weeks, we found major, previously unreported issues in these datasets.

Today, we're sharing our findings and open sourcing manifests for 4 of the most commonly used datasets. Shoutout to @sambhavgupta159 for doing amazing work here

[carlo agostinelli @carloagostinel2]

It’s been incredible watching @calixo888, @Jannikschg, and the entire @PantheonInc team build over the past few months.

Their level of ambition is rare.

We’re incredibly grateful to be working so closely with this team and even more grateful to call them friends.

Excited for everyone to see what’s coming.

[finbarr @finbarrtimbers]

Great to see more open data quality work. Data quality work doesn't get nearly enough attention in the open source/weights community, but it's almost all of the work in industry.

[Kunvar Thaman @kunvar_]

The unreasonable effectiveness of actually looking at data

[Cathal Canavan @CanavanCathal]

two old friends, two frighteningly ambitious gentlemen.

v cool to see what @Jannikschg, @calixo888 and the Pantheon team are building!

[Asher Spector @amspector100]

the pantheon folks have a lot of grit and a lot of heart. very excited by what they're working on!

[Xiao-ke (XK) @xklionfish]

@amspector100 Appreciate you Asher!!! 🫶

["Leigh Marie" Braswell @LM_Braswell]

You can just annotate things

Important work from @PantheonInc ! If the past decade in AI has taught us anything, it’s to never underestimate the power of a great data pipeline.

[Mamoon Hamid @mamoonha]

Great work by the @PantheonInc team exposing critical flaws in public robotics datasets. Better robots start with better data. Publishing these findings and annotations helps the whole field move forward.