Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams
The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching.
Key changes:
➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed.
➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts.
➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days.
➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting.
Key takeaways:
➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k.
➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort.
➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
ok hear me out: a wearable auditory stim soundboard that plays things like FAHHH out loud when you hit a button
i think i could do numbers on brainrotted gen Z’s with the right marketing
I feel like electron apps aren't inherently bad, you guys just want to build native apps in javascript but no OS will support that so you have to ship a whole ass browser as a compatibility layer
Announcing Bagel-Bench: the definitive guide to NYC bagels.
Every Friday for the past 2 years, we’ve ordered bagels for the whole @ExtendHQ team. 1000s of bagels later, we’re publishing our findings:
S-tier: Apollo and Bagel & Schmear are SOTA, pushing the frontier on the taste × reliability Pareto curve. two of NYC’s leading applied bagel labs
A-tier: Brooklyn Bagel, NYC Bagel & Coffee House, and Murray’s are strong production-ready alternatives. The gap to the frontier is real, but we remain bullish.
B-tier: solid baselines, but we’d like to see more bagel R&D
C-tier: these shops have impressive distribution. We’d love to see that same investment in the core model
D-tier: significant work to do before we’d recommend consumption
like all benchmarks, this one is biased. unlike most benchmarks, we’re admitting it upfront.
what bagel should we eval next?
You can either hold crypto and keep being stressed out if a math breakthrough would drain your wallet; or someone stealing your keys would drain your wallet; or someone kidnapping you and forcing you to hand over your keys would drain your wallet
Or you can just not hold crypto