Andy Zeng

Co-founder @GeneralistAI general intelligence born from the physical world
✗DMs → email

[Andy Zeng @andyzengineer]

if you look closely at the reliability test later in the video,

you’ll see a few examples of “physical reasoning” -- it tries something, it doesn’t fit (because physics), so it backtracks, switches gears, then does something entirely different to reach the goal.

in this case, it’s taking out the stuck puck, making space elsewhere, and putting it back in.

things like this (… a sort of physical problem-solving with step-by-step “chain-of-actions”) should just work with models out of the box.

[Generalist @GeneralistAI]

How many pucks could a puck bot pack if a puck bot could pack pucks? 🏒

12 hockey pucks is a tight fit in this container – watch the model fix the arrangement when pucks get stuck or misplaced.

GEN-1.5 doesn't just puck around. Read more about it in our blog post and sign up for the waitlist in the comments below.

[Ben Barry @benbarry]

Designed a new website for Generalist.

https://t.co/lXSY3RfHZc

[Generalist @GeneralistAI]

GEN-1.5 puts together a kit of cosmetics, then puts together another kit, then another kit, then another...

We're expanding access to GEN-1.5
Sign up to get in line
https://t.co/kaM93SJkXM

[Generalist @GeneralistAI]

We're expanding access to GEN-1.5

Sign up to get in line
https://t.co/kaM93SJkXM

[Evan Morikawa @E0M]

2 years ago, I personally spent dozens of hours trying to get this task to work. It was our first “too hard” task. Now it’s just another day here.

[Lucas Beyer (bl16) @giffmana]

Mesmerizing!

Also, *exactly* 10y ago i was at a startup with exactly these arms and very similar grippers with exactly this camera placement.

But models from 10y ago: cnn + lstm, no real pretraining.

What i see now, from multiple startups, it definitely a big step up.

[Generalist @GeneralistAI]

GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another...

Read more about GEN-1.5 in our blog post in the comments below.

[Corinne Marie Riley @CorinneMRiley]

Feels like every week in robotics there’s a new ‘this is the GPT-3 moment for robotics 🤖’ announcement. 

We brought on @GeneralistAI CEO & Co-Founder @peteflorence on @GreylockVC Change Agents to dig into what’s going on at the frontier of robotics models.

We covered the company’s latest Gen 1.5 model, few-shot learning, training robots on different embodiments, and the milestones towards a more generalized physical model. 

Timestamps:
00:51 The inception of Generalist
04:13 Long-term goal of the company
05:47 Parallels and differences between robotics and language models
12:05 Key differentiation in 1.5 Gen model
12:48 Robot vs banana
14:49 Emergent capabilities not explicitly trained for
17:13 Cross-embodiment and the importance of hands
22:26 Research vs working with customers
24:58 Beyond VLA vs world model
28:50 Future-looking milestones

Some of the top takeaways:

- One-shot and few-shot learning emerged without being trained for. Gen 1.5 can learn a new task from a single demonstration, and Pete compares it to the GPT-3 moment in language. In one example, a robot taught to sweep a cube into a bowl with a brush used a banana instead. Given a dustpan, it held the pan with one hand, swept with the other, then tipped the cube into the bowl. Neither behavior was explicitly trained, and Pete sees this as a signal of where the model's generalization is headed.

-Generalizing to new hands remains a challenging problem for cross-embodiment. Physical hardware doesn’t stay static, and so cross-embodiment - the ability of a physical AI model to adapt to different hardware systems - is vital for success. 

-Customer deployments are a valuable source of research inputs. Generalist actively partners with their customers for feedback, which they use to inform their research and make real-world evaluations.

-Generalist doesn't think in terms of "VLA vs. world model." Pete helped create early VLAs and has worked on world models, but he argues the goals matter more than the label, and the team is trained to think in a first-principled way when considering new research directions.

Watch the full episode at the link in the comments. Thank you to @peteflorence for joining us!

[Andy Zeng @andyzengineer]

post-training to hill-climb dexterity

[Generalist @GeneralistAI]

Robot puts a ring on it

GEN-1.5 puts an O-ring on a hydraulic rod gland, then takes the ring off, then puts the ring back on, then takes the ring off, then ring on, ring off, ring on...

Read more about GEN-1.5 in our blog post in the comments below.

[Generalist @GeneralistAI]

A simple example of physical prompt steerability: same environment, different prompts, different behaviors.

Thanks @JagdeepBhatia8 for the suggestion.

Read more about GEN-1.5 in our blog post in the comments below.

[Generalist @GeneralistAI]

We've reduced the time it takes to go from physical prompt → robot behavior.

The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work.

Read more about GEN-1.5 in our blog post in the comments below.

[Will Knight @willknight]

I'm usually wary about talk of "the ChatGPT moment for robots" because language is less complex than the physical world. After visiting @GeneralistAI, though, where I saw their robots perform impressive manipulation with just a prompt, I think you can see it happening in a limited way. (Thanks to @peteflorence and co for having me!) https://t.co/VdsK6jE7rS

[Pete Florence @peteflorence]

Good question thanks Ani. Here's a video that gives a sense of the answer here about the effect of prompting:

- in the first part, the model is just babbling with no prompt in context
- at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)

[Ben Barry @benbarry]

We started planning yesterday’s @generalistai GEN-1.5 announcement about 3 weeks ago. Two weeks ago, already feeling whiplash from new results, I asked ChatGPT to create this image to share in our internal meme channel.

[Ben Barry @benbarry]

I remember having a conversation with @ilyasut in the early days of @OpenAI (~2018) about how the pace of progress in AI research can be surprisingly fast.

Overnight, things can go from feeling nearly impossible to just working.

I felt it then, and I feel it now.

[himanshu @himanshustwts]

absolute GPT-3 moment for robotics
holy moly

[Pete Florence @peteflorence]

Ever since I started working on robot foundation models a handful of years ago, the broad ability to one-shot in-context learn has been the single most vivid goal in my mind along the long road ahead. It’s something @andyzengineer and I have especially been thinking together on for several years now, and something shared by the whole team at Generalist as a major goal.

But even before anybody started talking about robot foundation models, too, the capability this enables, i.e. generalized one-shot learning, has been both incredibly concrete and elusive. In terminology that I think many other people have come up with as well, in grad school we used to talk about a “Ctrl-C-Ctrl-V” type of capability. See something, and have the robot do it too. This same type of idea inspired the name of the 1970 MIT copy demo (https://t.co/qs08JRYJGl).

The thing is, it sounds simple, but is incredibly hard since the world is never quite the same when it got copied and where you want to paste it. The real world can be hard to predict and is full of variation. To do this, you need strong generalization, and it needs to be acquired in a single example. Doing this over a wide range of tasks, especially for dexterous tasks, is hard mode.

Lots of the components of the idea of making this all happen have been there for a long time. As an example, this 2017 NeurIPS paper “one-shot imitation learning” https://t.co/IWr9KWeeGi has excellent vision, with ambition well beyond what was achievable at the time, and although it’s not referred to as “in-context learning” since it was pre-Transformer, it actually uses attention to condition on a single demonstration.

And now, many things have happened since early 2017, including the broadly celebrated arrival of one/few-shot in-context learning in language models in 2020.

This new model GEN-1.5 takes in everything we have built and learned over the past couple years at Generalist. It has been training for 8 months. It has taken an incredible amount of commitment and grit from the whole team to get here. The level to which this model has survived many surgeries has continued to surprise me. And its capabilities have continued to surprise as well. We found compositional generalization on Friday. We found sim2real prompting earlier last week. We filmed the contiguous uncut videos of live prompting yesterday.

To be clear, the success rates are modest, and there’s still a long way to go. But now I have definitely seen a ~decade-long imagination come into the real world.

[Ben Barry @benbarry]

It feels like almost every day at this company there is another significant result! The speed of progress makes my job of documenting and communicating what we are doing a challenge, but it's also so much fun!

[berkay @BerkayAntmen]

Our latest pre-training comes with zero-gradient / in-context learning! So crazy to witness and glad we can talk about it publicly now.

[Andy Zeng @andyzengineer]

RT @felixwyw: At 10:06pm on Aug 3, I watched a robot do something I thought was years away.

We were working on few-gradient learning with…