Embodied Analysis

Unified Eval for Embodied Intelligence | [email protected]

[Embodied Analysis @embodianalysis]

Embodied Analysis brings physical-agent and world-model evaluation into one framework, with standardized tasks, harnesses, and environments.

Today, we’re introducing the platform with two evaluation tracks:

➤Physical agents: Can they control robots and other physical embodiments to complete complex, long-horizon tasks?

➤World models: How well do they satisfy physical plausibility, goal completion, and instruction following?

We evaluate across diverse tasks in economically valuable domains. By holding evaluation conditions consistent across models, we aim to make their capabilities and limitations easier to compare.

The framework is organized around three questions: What can the model do? Under what conditions? Where does it fail?

More on the methodology in the thread below.

Which task or failure mode would you want tested before trusting an embodied AI model in your workflow?

[Embodied Analysis @embodianalysis]

Claude Opus 5.5 announcement describes the capability and efficiency gains last month。We examined whether those gains extend to simulated robot tasks. Across 16 direct comparisons in the same Embodied Analysis experiment, Opus 5.5 was preferred 14 times and Opus 5 twice. Neither model visibly completes either of the two tasks shown here.

➤ Three bananas in the bin: Both start with two bananas already inside, so only one needs to be added. Both end with two still in the bin.
➤ Mug under the coffee dispenser: Opus 5.5 leaves the mug on the counter in front of the drip tray. Opus 5 moves it away from the dispenser. Neither clip shows the required placement.

Opus 5.5 was preferred in both pairs. The banana task makes the distinction especially clear: different movements and a preference for the newer model, but the same final count.

Preference and task completion measure different things. The 14 vs 2result shows which attempts annotators preferred; it does not establish higher completion rates or improved efficiency in robotics.

[Embodied Analysis @embodianalysis]

We hope you've found this thread helpful.

Follow us @embodianalysis for more.

Like/Repost the quote below if you can:https://t.co/7ej1Khj8zT

[Embodied Analysis @embodianalysis]

Claude Opus 5.5 was preferred over Opus 5 in 14 of 16 direct comparisons in our simulated robotics evaluation.

Here're two example tasks that reveal their behavior differences:

➤ Croissant and jam: Opus 5.5 places the croissant on the cutting board and moves on to the cabinet. Opus 5 leaves the croissant on its plate. Neither places the jam.
➤ Microwave: Opus 5 was preferred in this pair. It reaches toward the lower Start button area, while Opus 5.5 works near the upper controls. The clips do not clearly confirm that the microwave started.

A completion only score would give both croissant runs the same outcome: unfinished. It would miss the difference in progress through the instruction. The 14/2 preference result captures a relative judgment, not a task success rate.

[Embodied Analysis @embodianalysis]

We hope you've found this thread helpful.

Follow us @embodianalysis for more.

Like/Repost the quote below if you can:https://t.co/7ej1Khj8zT

[Embodied Analysis @embodianalysis]

In our Oct 17 robotics preference vote snapshot, Gemini 3.8 Flash leads GPT-6 Astra 9/4. In these red mug runs, both leave the cup with its opening facing up.

We asked each model to turn the mug upright.

➤ Gemini nudges and turns the mug along the tabletop until it settles upright.
➤ Astra lifts and releases it; the mug tips before settling upright.
➤ GPT-6 Sol moves a nearby white cup, while the red mug ends on its side.
➤ Claude Opus 5.5 keeps adjusting near the red mug, but its gripper obscures the opening in the final frame.

Gemini also leads Sol 10/4, but trails Opus 5.5 5/10 across the wider vote set. Those are preferences across tasks, not success rates for this clip. The footage shows one run per model at 1.25×, with three panels using closing excerpts.

[Embodied Analysis @embodianalysis]

@ZiqiPang Good question. In the croissant task, it placed the croissant on the board while Opus 5 left it on the plate, though neither completed the full task. These runs show the performance difference, but not what caused it.

[Embodied Analysis @embodianalysis]

We hope you've found this thread helpful.

Follow us @embodianalysis for more.

Like/Repost the quote below if you can:https://t.co/7ej1Khj8zT

[Embodied Analysis @embodianalysis]

Muse Wonder ranks #6 on Embodied Analysis Index.

On the Physical Agent Capability Index, Muse Wonder has 1034 Elo, while GPT-6 Astra ranks #1 with 1349 Elo.

In 10 direct comparisons with GPT-6 Astra in the same four-model batch, Astra was preferred 8 times and Muse twice. This folding task was one of Muse's two preferred runs, although neither run ended with a neatly folded shirt.

➤ Muse Wonder draws the shirt inward, leaving it loose and wrinkled.
➤ GPT-6 Astra begins handling it earlier and regrips repeatedly, leaving the fabric bunched on the table.

The vote identifies the preferred attempt, not a completed fold. Preference and task completion should be reported separately.

[Embodied Analysis @embodianalysis]

We hope you've found this thread helpful.

Follow us @embodianalysis for more.

Like/Repost the quote below if you can:https://t.co/7ej1Khj8zT

[Embodied Analysis @embodianalysis]

Gemini 4 Argon “closely follows” GPT 6 Astra on our Physical Agents Index. As of October 14, Argon ranks second of 18 models at 1,324 points, 25 behind Astra's 1,349.

Here's an example run on the same task: put four bottles into the bin, using handovers when needed. Both runs start from the same layout.

➤ Argon releases all four bottles over the bin within its 69 second recording. The final white bottle is still falling when the footage ends.
➤ Astra's 99.5 second recording ends with the yellow and green bottles still on the table. It releases the pink and white bottles earlier.

Both make repeated gripper adjustments. Argon's white bottle also returns to the table mid-run before a later release. The contrast is in the final object states, not an absence of retries.

One run per model, not a success-rate estimate. Both full recordings are shown at 6× speed; durations above refer to the original footage.

[Embodied Analysis @embodianalysis]

@jensen_kimmit Indeed.We’ll keep testing their models on robotics tasks and tracking their progress.

[Embodied Analysis @embodianalysis]

@NVIDIARobotics Separating reconstruction, grounding, and policy learning should make the results easier to interpret. Looking forward to seeing which gains carry through to robot performance.

[Embodied Analysis @embodianalysis]

@Figure_robot Thanks for sharing the longer footage. It gives people more room to inspect pauses and retries than a short highlight reel.

[Embodied Analysis @embodianalysis]

@SkildAI The discussion of speed and accuracy in the writeup is useful. Seeing task completion rates alongside cycle times would make these deployments easier to compare.

[Embodied Analysis @embodianalysis]

@BostonDynamics @HMGnewsroom Looking forward to seeing how Atlas handles different parts and workstation layouts during training.

[Embodied Analysis @embodianalysis]

@NVIDIARobotics The subtask breakdown looks useful. An overall success rate can hide whether a policy struggles with the grasp, placement, or a later step.

[Embodied Analysis @embodianalysis]

@agilityrobotics @FortRobotics Connecting the robot to the facility's existing safety systems makes sense. I'd be interested in how a task resumes after a safety stop.

[Embodied Analysis @embodianalysis]

@UnitreeRobotics Curious how the backdrivable joints respond when an object starts slipping during a grasp.

[Embodied Analysis @embodianalysis]

@Figure_robot Curious which types of human video contribute most to robot performance. The task rebalancing step in the writeup seems worth watching as the dataset grows.

[Embodied Analysis @embodianalysis]

@TomAuto_ We see that loop too. We’ll take a closer look.

[Embodied Analysis @embodianalysis]

We gave GPT-6 Astra and Claude Opus 5 the same task in a simulated robotics environment: play tic-tac-toe with an opponent and fill all nine cells.

Below is one run per model, shown at 2× speed. All timestamps refer to video playback.

➤ Completion: Astra’s board has all nine cells filled and its arms are back at rest by approximately 20 seconds. Opus 5 is still working on an incomplete board when the clip ends.
➤ Pickup accuracy: Both successfully grasp pieces. In these runs, the more visible difference is in completing placements.
➤ Adjustment: Opus 5 repeatedly repositions during later moves. Astra also makes adjustments before its final piece settles.
➤ Motion efficiency: Astra follows a more regular pick–place–retract sequence, while Opus 5 spends longer approaching and realigning around the board.

[Embodied Analysis @embodianalysis]

@JustinLin610 argon's robotics capability is closely following astra

https://t.co/WgVsBynSBe

[Embodied Analysis @embodianalysis]

Gemini 4 Argon “closely follows” GPT 6 Astra on our Physical Agents Index. As of October 14, Argon ranks second of 18 models at 1,324 points, 25 behind Astra's 1,349.

Here's an example run on the same task: put four bottles into the bin, using handovers when needed. Both runs start from the same layout.

➤ Argon releases all four bottles over the bin within its 69 second recording. The final white bottle is still falling when the footage ends.
➤ Astra's 99.5 second recording ends with the yellow and green bottles still on the table. It releases the pink and white bottles earlier.

Both make repeated gripper adjustments. Argon's white bottle also returns to the table mid-run before a later release. The contrast is in the final object states, not an absence of retries.

One run per model, not a success-rate estimate. Both full recordings are shown at 6× speed; durations above refer to the original footage.