WorldOracle Bench

Benchmarking Action-Conditioned World Models against Real-World
Executions.

[WorldOracle Bench @WOB_WorldOracle]

For Cosmos 3 Nano, adaptation starts with the action interface.

We replace the pretrained 29D AgiBot action interface with a 20D dual-arm representation: relative translation, relative rotation, and gripper state for each arm.

[WorldOracle Bench @WOB_WorldOracle]

We adapt and evaluate Cosmos 3 Nano in WorldOracle.

We initialize it from the official 16B checkpoint and compare its predicted rollouts with paired real-world executions.

#NVIDIACosmos #PhysicalAI

[WorldOracle Bench @WOB_WorldOracle]

Cosmos evaluation includes physical state correctness and interaction spatial alignment.

WorldOracle's metric case studies show why both matter:
Aligned action, incorrect physical outcome.
Correct physical outcome, misaligned action.

🔗Link:https://t.co/Cx14KNth1R

[WorldOracle Bench @WOB_WorldOracle]

These comparisons are scored along four complementary dimensions of execution coherence.

In a study with 128 prediction pairs and 119 participants, all four metrics achieve 88.1-93.2% overall agreement with human preferences.

🔗Link:https://t.co/Cx14KNth1R

[WorldOracle Bench @WOB_WorldOracle]

Does the predicted future match reality?

Our benchmark WorldOracle evaluates action-conditioned world models against paired real-world executions through three stress tests:

1. Causal obedience
2. Distribution transport
3. Temporal endurance

#PhysicalAI

[WorldOracle Bench @WOB_WorldOracle]

To run these stress tests, we replay recorded actions open loop, including successful and failed attempts.

We test across AgiBot G2, YAM, and human-hand embodiments and a G2 policy-rollout split, then track execution coherence over long, multi-segment rollouts.

[WorldOracle Bench @WOB_WorldOracle]

Multi-View Action Coherence evaluates predicted robot motion against paired real-world execution across synchronized views.

Robot-arm alignment is scored independently in each view, then the scores are averaged. Video: AgiBot G2 real execution.

🔗Link:https://t.co/Cx14KNth1R

[WorldOracle Bench @WOB_WorldOracle]

WorldOracle is a benchmark for evaluating action-conditioned world models (ACWMs) against paired real-world executions.

Given the same initial observation and low-level action trajectory, does the predicted future match reality?

🔗Link：https://t.co/Cx14KNth1R

[WorldOracle Bench @WOB_WorldOracle]

Same initial observation. Same action trajectory.

In the "pick fruit" example, the real gripper misses the fruit, but the model hallucinates a successful grasp.

WorldOracle evaluates execution coherence: whether the predicted rollout matches its paired real-world execution.

[WorldOracle Bench @WOB_WorldOracle]

The ACWM replays the recorded action trajectory open loop. WorldOracle evaluates the predicted rollout against its paired real-world execution using:

Physical State Correctness
Interaction Spatial Alignment
Key Interaction Area Fidelity
Multi-View Action Coherence

[WorldOracle Bench @WOB_WorldOracle]

Hello, X! 👋
We are WorldOracle Bench, dedicated to evaluating frontier AI’s ability to understand the world, reason about reality, and predict the future.
Follow us as we explore the boundaries of AI capabilities.
#AI #Benchmark #WorldOracleBench