haarala

ai product partnerships x automationbench @zapier

[Elon Musk @elonmusk]

Cool that Grok 4.5 is #1 in some respects, even with respect to Fable 5

[Artificial Analysis @ArtificialAnlys]

SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules

AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination.

Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails.

Key takeaways:

➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks

➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49)

➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing

➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh)

➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5)

➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%)

Congratulations to @SpaceXAI and @elonmusk on topping the leaderboard!

[Artificial Analysis @ArtificialAnlys]

Announcing AutomationBench-AA, our independent leaderboard for Zapier’s AutomationBench, testing whether AI agents can automate real SaaS workflows while adhering to business rules

We partnered with @zapier to run AutomationBench-AA on their private benchmark subset. This benchmark is a complex agentic workflow automation test across simulated SaaS applications. Models must complete 657 tasks spanning Finance, HR, Marketing, Operations, Sales, and Support, working across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot.

Unlike Zapier’s hosted leaderboard, the headline score for AutomationBench-AA shows the share of objectives a model completes without violating any guardrails. Claude Fable 5 and Opus 4.8 from @AnthropicAI lead with scores of 48.6% and 48.5%, followed by @GoogleDeepMind's Gemini 3.5 Flash at 42.6% and @OpenAI's GPT-5.5 (xhigh) at 42.1%. With Anthropic’s new classifier, Fable 5 fell back to Opus on ~18% of tasks.

Key elements of AutomationBench:

➤ Real workflow patterns, simulated environments: Tasks are drawn from real workflow patterns on Zapier and run in simulated SaaS environments, where a single task may span a range of applications like CRM, email, calendar, and messaging platforms.

➤ Autonomous API discovery: Models interact with each app through REST APIs, discovering the endpoints they need through structured tool calls and navigating environments with irrelevant and sometimes misleading records.

➤ Objectives and guardrails: Models are scored against nearly 12,000 assertions Zapier built to test that the model completed the task correctly in full. Each assertion is classified as either an objective the agent must achieve, or a guardrail that already passes initially and must not be broken.

➤ Programmatic environment grading: Tasks are graded solely on whether the correct data ended up in the right systems, with deterministic checks against the environment. Each task runs once with a 50-turn cap.

Key results for AutomationBench-AA:

➤ Claude Fable 5 (max) leads at 48.6% but falls back to Opus 4.8 in ~18% of tasks. It completes 73% of task objectives, with the fallback behavior likely explaining the limited uplift compared to Opus.

➤ Every model breaks business rules: Guardrail violations range from 0.46 per task (Gemini 3.5 Flash) to 1.26 (Qwen3.7 Plus). Gemini 3.5 Flash completes 15.0 objectives per guardrail violation, the best ratio of any model, ahead of Claude Opus 4.8 (max, 13.5).

➤ Gemini 3.5 Flash performs well for its price: At 42.6% and $0.49 per task, it effectively matches GPT-5.5 (xhigh, 42.1%, $1.32 per task) at ~37% of the cost.

➤ GLM-5.2 (max) from @Zai_org is the leading open weights model at 27.8%. This places the open weights frontier ~10 points behind Gemini 3.1 Pro Preview, and with substantially higher guardrail violations per task.

➤ Finance workflow tasks are the most difficult to automate today: across the models we evaluated at launch, agents complete roughly half the proportion of objectives on Finance tasks, compared to Support and Operations tasks.

We would like to thank Zapier and the benchmark authors for their great work developing this evaluation for important SaaS workflows, and appreciate their collaboration in launching AutomationBench on Artificial Analysis!

[haarala @mhaarala]

@joewill1082630 @OfficialLoganK @zapier re: translation beyond benchmarks: the tasks are modeled on real business workflows we've seen over ~15 yrs. the gap between benchmark task and real workflow is intentionally small - so, the cost advantage should translate pretty well

[haarala @mhaarala]

@joewill1082630 @OfficialLoganK @zapier it excelled at persistence and tackling at Operations (20%) and HR (19%) workflows, the domains with the most step coordination/strict policy adherence - but struggled w/ strict output formats and making decisions based on math it has to do on its own

[haarala @mhaarala]

@fsfarimani @OfficialLoganK @zapier yes! fully open. white paper/methodology/leaderboard are public at https://t.co/LdSzvzc0HO and https://t.co/rYehtuPTvd. binary pass/fail scoring across 6 domains, 47 real app schemas. @lukasb + @danielwshepard + i are happy to answer any questions on the approach

[haarala @mhaarala]

@ilyasturki_ @OfficialLoganK @zapier we're measuring something most benchmarks don't: whether a model can finish real, multi-step business workflows end to end. intentionally hard (top score is 14.5%) because that's the gap that actually matters in production/the real world. methodology here: https://t.co/rYehtuPTvd

[haarala @mhaarala]

@trifon_getsov @OfficialLoganK @zapier rooting tasks in reality was a core tenet of building AutomationBench. tasks are generated from real workflow patterns we've seen on Zapier over ~15 years, esp. the ones businesses actually try to automate and struggle with -- whitepaper has addl details: https://t.co/rYehtuPTvd

[haarala @mhaarala]

@chontang @OfficialLoganK @ashutosh_270497 @zapier very helpful feedback -- we launched AutomationBench about ~a month ago, and are actively working to improve + expand on what we measure/report in the leaderboard. in the meantime, open source models can be run against AutomationBench on Prime Intellect: https://t.co/xi4AmB8Fuo

[Logan Kilpatrick @OfficialLoganK]

Gemini 3.5 Flash ranks #1 on Automation Bench (from Zapier), beating every other frontier model at a much lower cost

[Wade Foster @wadefoster]

We built an AI benchmark that measures real work.

Today we're releasing it to everyone.

AI evals tell you whether a model can do complex reasoning or generate code. Useful, but usually not the question our customers ask. They want to know: can this model find the right CRM record, send the right follow-up, and not break anything along the way?

We went looking for a benchmark that tested that. Nobody had built one, so we did.

@Zapier’s AutomationBench drops AI models into realistic business environments across six domains (Sales, Marketing, Ops, Support, Finance, HR) and checks whether the work actually got done.

The tasks include live CRM data, inbox threads with ambiguous context, and multi-step tool chains where one wrong call cascades.

Scoring is deterministic: either the right records were updated and the right messages were sent, or they weren't.

It’s useful enough that we're releasing it publicly today. Open task set, open methodology, open leaderboard. Everyone should have access to this.

No model has cracked 10%. Yet.

Try it here: https://t.co/V7qHAGX7Ql

[haarala @mhaarala]

@eddiejiao_obj @cmmnknwledge would love to try!

[haarala @mhaarala]

@thebirdisntreaI lmaoooooo GOT EM

[haarala @mhaarala]

@thebirdisntreaI i had a visceral reaction reading this

[haarala @mhaarala]

@thebirdisntreaI it’s not HBO MAX it’s just MAX get it right dummy

[haarala @mhaarala]

@DocumentingATX really appreciate you -- the only source of info we've got to keep our family safe

[haarala @mhaarala]

@thebirdisntreaI you can say anything on everyone’s favorite app X!

[haarala @mhaarala]

@davidrisher @onowo_ @RoyaBacklund @palashp40616755 @lyft @AskLyft this is the same response you’ve copied and pasted to tens of other tweets… be real at least

[palash pandey @palashp40616755]

@AskLyft My Lyft driver drove off with my pet cat still in the car.

I was taking my cat to a vet appointment, I was sitting behind the driver and had the cat carrier on the floor of the passenger side back seat.

[haarala @mhaarala]

@KUTnathan @KUT westgate @ stassney, westgate @ jones, westgate @ western trails, lamar @ ben white… westgate traffic rarely stopping at any of those