Evan Morikawa

VPE @Generalistai. GP ZSF. Prev: led eng at @openai. Launched & scaled ChatGPT, & APIs. @OlinCollege alum.

[Evan Morikawa @E0M]

Our mid-term vision is a world w/ billions of robots. Some will be humanoid, but we believe there will be a Cambrian explosion of form factors. These robots will be run by thousands of startups & businesses in every industry. All powered by a @GeneralistAI brain. It starts today

[Generalist @GeneralistAI]

We're expanding access to GEN-1.5

Sign up to get in line
https://t.co/kaM93SJkXM

[Evan Morikawa @E0M]

2 years ago, I personally spent dozens of hours trying to get this task to work. It was our first “too hard” task. Now it’s just another day here.

[Generalist @GeneralistAI]

GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another...

Read more about GEN-1.5 in our blog post in the comments below.

[Evan Morikawa @E0M]

@Evan_Tao_T @ieeeiros @Evan_Tao_T would love to chat about this. DM me?

[Evan Morikawa @E0M]

@JacobZietek Congrats!

[Evan Morikawa @E0M]

@GergelyOrosz Very excited for this article

[Generalist @GeneralistAI]

A simple example of physical prompt steerability: same environment, different prompts, different behaviors.

Thanks @JagdeepBhatia8 for the suggestion.

Read more about GEN-1.5 in our blog post in the comments below.

[Pete Florence @peteflorence]

Good question thanks Ani. Here's a video that gives a sense of the answer here about the effect of prompting:

- in the first part, the model is just babbling with no prompt in context
- at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)

[Will Knight @willknight]

I'm usually wary about talk of "the ChatGPT moment for robots" because language is less complex than the physical world. After visiting @GeneralistAI, though, where I saw their robots perform impressive manipulation with just a prompt, I think you can see it happening in a limited way. (Thanks to @peteflorence and co for having me!) https://t.co/VdsK6jE7rS

[Ben Barry @benbarry]

We started planning yesterday’s @generalistai GEN-1.5 announcement about 3 weeks ago. Two weeks ago, already feeling whiplash from new results, I asked ChatGPT to create this image to share in our internal meme channel.

[Evan Morikawa @E0M]

@unixpickle @GeneralistAI Classic sim is rough for dexterity. Imagine peeling an orange in sim. Then it comes down to capital and compute efficiency. Big differences across data sources here.

[Ben Barry @benbarry]

I remember having a conversation with @ilyasut in the early days of @OpenAI (~2018) about how the pace of progress in AI research can be surprisingly fast.

Overnight, things can go from feeling nearly impossible to just working.

I felt it then, and I feel it now.

[Felix Wang @felixwyw]

At 10:06pm on Aug 3, I watched a robot do something I thought was years away.

We were working on few-gradient learning with GEN-1.5, wondering how far we were from landing physical prompting: show the robot a task once, and it just does it. We YOLOed it, and the robot imitated exactly what we demonstrated, with 0 fine-tuning. We gave more prompts for different tasks. More successful rollouts landed. I could not believe my eyes.

Looking back, this represents the culmination of a huge amount of work across the entire team, many failed experiments, and years of groundwork from the community. I just happened to be the one standing in front of the robot when it all clicked.

@andyzengineer once asked me: "If Generalist failed tomorrow, what would we want to have left behind that could benefit humanity?"

I believe we found one answer: an existence proof of physical prompting.

[Ted Xiao @xiao_ted]

In-context learning for robot control has been a holy grail for many years. GEN-1.5 is the best result on this frontier I have ever seen!

Feels like a step change moment; getting goosebumps reminiscent of the first times I played with GPT-3.

[Sholto Douglas @_sholtodouglas]

@GeneralistAI GPT3!

[Evan Morikawa @E0M]

The GPT-3 moment is here

[Generalist @GeneralistAI]

Introducing GEN-1.5, a one-shot learner.

It can learn new tasks in a few seconds. Show it what to do, and it generalizes.

This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.

[Lewis Jones @thelcjones]

You should take 30 seconds to compare how much smoother and more dexterous @GeneralistAI is compared to Google’s model from last week (e.g. clapperboard video). They’re on another level and word on the street is that they have insane researcher retention, kind of like how Anthropic is a black hole of talent in LLMs where the best researchers all eventually end up there.

[Ben Barry @benbarry]

I wish we had done a better job communicating this, because it's such a mind-bending result!

Imagine you're reaching to pickup an object, and instantly your hand completely changed shape. You'd probably freak out! GEN-1 just instantly adapts and accomplishes the task. 🤯

[Jacob Zietek @JacobZietek]

generalist's hardware team doesn't get enough credit for their creativity. compliance is in the tool mount. qdd-like contact tolerance, harmonic drive precision, and the deflection is readable on camera because of how it's shaped and colored. no f/t sensor per tool, so cool

[Evan Morikawa @E0M]

Thoughts on high-DoF 5-finger hands

I believe a perfect robot replica of the human hand could have complete overlap w/ the space of human potential

However I think this undersells the potential of robotics & ignores the practicalities of early deployment and its benefits

One key idea is that high DoF hands are not zero sum with specialized tools when you have a general purpose model. This is not an either or. This is now a "yes and" situation. The full space of human potential is actually quite limited. Why are we only striving for this?

Doing this prematurely has a lot of issues. Current 20-some DoF 5-finger robotic hands are VERY expensive. ~$100k for a pair. They break all the time. They're hard to repair. When you care about unit economics or uptime, they make zero sense today.

That being said, they are getting cheaper and more robust quickly. I believe you'll see economically-viable, robust-enough hand hardware in a matter of years. Those years matter a lot right now though.

This would be like pitching self-improving coding agents 5 years ago when we had GPT-3. It would have been a different world if no one deployed LLMs early and often.

While there are many tasks that are better suited to a high-DoF hand, there are many more that are better suited to different form factors. The home consumer space clearly can benefit from high DoF hands, but for every robot that will be in the home, there will be dozens more in industry and in our fields doing a myriad of repetitive tasks. It seems clear that in those regimes some degree of specialization is valuable. It's likely this will outnumber even the consumer space.

High DoF hands, like humanoids, I think are a failure of the imagination. It's only the "known known". Much like how the real value of LLMs was everything we didn't expect, I believe most of the value of robotics will come from the "unknown unknowns". Innovations in other types of hardware will certainly be a part of this.

[Generalist @GeneralistAI]

Why create robot intelligence for just one hand, when we could have it learn from many?

GEN-1, our latest embodied foundation model, now supports a broad range of end effectors from 5-finger hands, to specialized tools, and everything in between.