Hamza

@openai codex ambassador, @rootlyhq AI Labs, undergrad researcher @uoft, @harvard

[Hamza @hamza72510]

We benchmarked GPT-6 models in DOOM by having LLM agents fight each other. Across 120 matches, we found:

- Astra had the highest win rate at 82.5%
- Sol made the fastest decisions at avg of 5.34s
- Luna had the most wins per dollar at 12.8

GitHub: https://t.co/dt3JHU7338
@OpenAIDevs @rootlyhq

[Hamza @hamza72510]

I ran Claude Opus 5.5 Max and GPT-6 Astra Max on LIBERO-LoHo, a long-horizon robotic manipulation benchmark, to see how well frontier models can control robots and complete multistep tasks zero-shot. Across 5 episodes per model per task, here are the results:

- Astra matched or beat Opus in task success with ~88% fewer output tokens.
- Astra cost 42% less ($3.91 vs. $6.75/run).
- Astra was 57% faster (14.8 vs. 34.2 min/run).
- Opus used 16% fewer calls and 20% fewer action steps.

Code + Details: https://t.co/maQg2z6G6F

[Hamza @hamza72510]

I ran Claude Opus 5.5 Max and GPT-6 Astra Max on LIBERO-LoHo, a long-horizon robotic manipulation benchmark, to see how well frontier models can control robots and complete multistep tasks zero-shot. Across 5 episodes per model per task, here are the results:

- Astra matched or beat Opus in task success with ~88% fewer output tokens.
- Astra cost 42% less ($3.91 vs. $6.75/run).
- Astra was 57% faster (14.8 vs. 34.2 min/run).
- Opus used 16% fewer calls and 20% fewer action steps.

Code + Details: https://t.co/maQg2z6G6F

[Hamza @hamza72510]

@Greg_GL_87 Checkout the details in the Leaderboard: https://t.co/JlxmOcTI9A

[Hamza @hamza72510]

GPT‑6.1 Sol is insanely efficient.

On Doom Agent Arena, it outperforms Astra medium by 22 percent points (85% v 63% win rate) at only 14% of Astra's total cost ($5.7 v $40.3)

On DeepSWE, OpenAI also reports 6.1 Sol medium achieving a slightly higher score than GPT‑6 Astra (73% v 72.8%), at also just 13% of the cost.

Is there any reason to still use GPT‑6 Astra?

[Hamza @hamza72510]

GPT‑6.1 Sol is insanely efficient.

On Doom Agent Arena, it outperforms Astra medium by 22 percent points (85% v 63% win rate) at only 14% of Astra's total cost ($5.7 v $40.3)

On DeepSWE, OpenAI also reports 6.1 Sol medium achieving a slightly higher score than GPT‑6 Astra (73% v 72.8%), at also just 13% of the cost.

Is there any reason to still use GPT‑6 Astra?

[Hamza @hamza72510]

@paw_lean @sama @thsottiaux @tejalpatwardhan @CoryDecareaux @romainhuet Congrats Pauline!

[Hamza @hamza72510]

Check out the latest results here! https://t.co/JlxmOcUfZ8

[Hamza @hamza72510]

@philip_mit Congrats!

[OpenAI @OpenAI]

10am PT, on the dot.

[Aquib @heystykid]

POV: you switched to a low reasoning effort model

[Dmytro Hrybov @dimentary]

hilarious to watch how Opus 5.5 designs a 3d asset, prints it, and controls a robo arm to take it out of the printer

[Hamza @hamza72510]

The setup was as follows:

> Astra: writes its own routes and firing policies, taking up 6.61s on average for valid action generation. It has more route flexibility but also more error prone (11.5% invalid plans)

> Jev: selects from 20 validated plans per decision, at 317ms average Inference.

Takeaway: fast selection and route planning are different capabilities. Astra can plan around by generating routes to counter Jev. by flanking and baiting.

Github: https://t.co/dt3JHU6vdA

[Hamza @hamza72510]

Can Astra’s planning outperform Jev’s fast decision making in Doom?

I put them head-to-head across 50 rounds:

Wins: Astra 33 v. Jev 17
Ave. Inference: Astra 6.61s v. Jev 317ms (~20.9× faster)
Cost: Astra ~42.30v.Jev~0.09 (~470× cheaper)

Jev made decisions quickly and won several early rounds. Astra won more overall and dominated the later rounds by adapting to Jev's decisions.

(Setup and gameplay in 🧵)
@OpenAIDevs @CompleteSkeptic

[Hamza @hamza72510]

Right, the goal isn’t necessarily to test realworld generalization. For embodied robotics, something like mujoco or a high fidelity physics sim would make sense.

The point of this project is to compare models in the same domain where we can evaluate the tradeoffs like latency, cost, planning quality, resource use, adaptation, and overall performance.

I did a deeper analysis on the previous models here: https://t.co/8ZT5htOCZL

[Hamza @hamza72510]

We benchmarked GPT-6 models in DOOM by having LLM agents fight each other. Across 120 matches, we found:

- Astra had the highest win rate at 82.5%
- Sol made the fastest decisions at avg of 5.34s
- Luna had the most wins per dollar at 12.8

GitHub: https://t.co/dt3JHU7338
@OpenAIDevs @rootlyhq

[Hamza @hamza72510]

It is interesting to see how different models explored.

From the current runs, attached is an heat map of how each model explored (executed routes). The Gray boxes are walls, star is shotguns and + is health-packs, meanwhile rows represent alternative spawns and columns models.

Astra shows the strongest concentration around shotgun and health-pack locations, which may help explain where some of its win-rate advantage comes from

[Hamza @hamza72510]

We run every model pair wise against every other model combination. So sol v. astra, sol v. luna, astra v. sol, etc.(with the sides reversed as well to account for any spawn/side bias).

Details on the methodology are here: https://t.co/8ZT5htOCZL

The setup is the same, but its an analysis on older gpt models.

[Hamza @hamza72510]

@adityajhalani43 Exactly, the goal isn't to find the 'best' model, It's to evaluate the tradeoffs and what each are good at in the same domain.

[Hamza @hamza72510]

Models learned strategies over time and adapted to the opponents actions. Eg. If the opponent eliminated a model from a certain route, the model would plan accordingly and in some situations bait or flank the next round.

Here is a write-up with older models I had benchmarked: https://t.co/8ZT5htOCZL

The setup is the same, just different models.

[Hamza @hamza72510]

@arjunaaqa @OpenAIDevs More models to come soon!

[Hamza @hamza72510]

@OpenAIDevs 🫡