Alex Vikati

Benchmarking your favorite LLM

[Alex Vikati @vikati]

@wsj's @eschisgall wrote that @typesafeai's Jev decision model is used by 25% of F500 and "sparks copycats". F500 already has dozens of credible alternatives today.

@michellechen etc shipped @cloudflare's clef and clef-flash

@awscloud shipped Strands Decider 2B

@databricks has a new ai_decide function

@perplexitydevs and @AravSrinivas shipped pplx-decider

Competitive advantages are really fleeting in this market. You can define a category and be surrounded by alternatives before your buyer reads the WSJ article.

And for buyers, more choices means a lot more homework. How do you figure out which is the right choice for your workload?

We are tracking these models on https://t.co/Qi7ocGCFMV

[@edwin 🌵📈 @edwin]

1/7

We asked OpenAI’s Dots to build an AI status dashboard: one place to check @claudedevs, @cursor_ai, @OpenRouter, @GeminiApp, @MistralAI, @cohere API health.

Twelve providers. Public page. Saved incident history. Private watchlists. Refresh every 15 minutes.

We didn’t specify the stacks. We repeated this experiment 8x.

[Alex Vikati @vikati]

Full report: https://t.co/eibB5F1NUF

[Alex Vikati @vikati]

Resend continues its impressive run in our latest Fable benchmark.

While custom/DIY rose to 21.4% overall and took ground in auth, caching, and feature flags, Resend defended the email category at 52%.

It’s a reminder that when you make your product/docs the right agent building block, you stay picked even as models get dramatically better at building things themselves.

[Zeno Rocha @zenorocha]

new Fable 5 benchmark is out!

they ran Claude Code 810 times, with no tool names in the prompts.

it would rather write code than install a dependency.

it hand-rolled auth 64% of the time. 71% for feature flags.

email? it picked @resend 52% of the time.

[@edwin 🌵📈 @edwin]

First raw volume. All-time, public GitHub:

Claude Code: 12.5M commits
Codex: 5.1M PRs
Copilot: 3.8M commits
Jules: 1.15M PRs
Cursor: 1M commits
Aider: 613k commits
Devin: 161K PRs

And those are floors. Private repos and squash merges never show up, so the real numbers are a lot bigger.

[@edwin 🌵📈 @edwin]

There’s almost no public data on how people actually use coding agents.

So we’re building an index by tracking agent activity on public GitHub repos (commits, PRs, etc).

Here’s what we found in our recent snapshot:

[@edwin 🌵📈 @edwin]

1/8

Mythos / Glasswing is clearly the main AI security story now: AI finding real vulnerabilities in existing production code.

For most teams, though, this question is more immediate:
Can an agent like Claude Code write a secure app in the first place?

[Alex Vikati @vikati]

The Claude Code leak shows a clear divide: select vendors gain small but compounding advantages.
Everyone else gets generic UX + install friction.

Direct impact on roadmaps and GTM.

[@edwin 🌵📈 @edwin]

1/9

The most interesting thing about the Claude Code leak for devtool companies:

Anthropic hardcoded 120+ vendor names across 7 different systems in the source. Anthropic explicitly included your tool name in the code (or they didn’t 🤷🏻‍♂️)

Thread 👇

[@edwin 🌵📈 @edwin]

After OpenAI/Astral acquisition announcement, we ran a benchmark on their tools.

Turns out Astral tools were already a core part of the Codex (and Claude Code) workflow for Python developers. Ruff + uv came out on top in 75% of cases for linting, packaging, and pretty much everything else.

Full report: https://t.co/633EpaIfTD

[@edwin 🌵📈 @edwin]

1) Winners & losers from our OpenAI Codex vs Claude Code Picks benchmarks

[Alex Vikati @vikati]

Past category winners relied heavily on marketing and sales, and GTM still matters to get in front of coding agents.

But agents won't keep using you unless you actually make them better (since the agents are trained with their harness).

That shifts long-term value toward real product quality, not just distribution.

[swyx @swyx]

every category leader in this list should be worth at least $5b btw because koding agents will be recommending them for the next 5 years + infra is stickier than agents

(full disclosure am smol resend angel)

[Alex Vikati @vikati]

The strongest argument against Taalas: AI models change every 3-6 months. Llama 3 → 3.1 → 3.2 → 4 in under a year. Each Taalas chip is locked to one model forever.

You're buying dedicated hardware for a moving target. The moment a better model ships, your chip is legacy silicon. Counter-argument: 2-month fab turnaround Taalas + 20x lower production cost means the refresh economics might actually work. If a model-specific chip generates 10x the revenue per watt during its window, the disposable hardware model could be net positive.

[@edwin 🌵📈 @edwin]

My friend @gentschev started using @Railway , @sentry and @resend for his https://t.co/O0eThSr3iD side project because of Claude Code's recommendations. The impact is real.

[Alex Vikati @vikati]

Coding agents like Claude are a massive new distribution channel for infra providers.

@Railway (~$124M raised) crushes peers like @render (~$258M raised) and @flydotio (~$115M raised) in Claude Code preferences.

Will agent recommendations drive outsized growth?

[@edwin 🌵📈 @edwin]

3/9 Deployment (Python)

Winner: @Railway – 82% picks. The dark horse: none of Vercel's hype machine, but quietly eating away tons of non-Next.js hosting from everyone (AWS, GCP, Heroku, Render, Fly, etc). When IPO?

Simplicity wins. @JustJake Railway is stealthily dominating.

[Alex Vikati @vikati]

Winner and losers from our Claude Code Picks benchmarks!

[@edwin 🌵📈 @edwin]

1/9 Winners & losers from our Claude Code Picks benchmarks that measure what Sonnet 4.5, Opus 4.5, and Opus 4.6 default to when building apps.

[Alex Vikati @vikati]

@zeeg @edwin @github @stripe @sentry @resend @zenorocha you have to send the dogs to @bugsnagstatus @rollbar https://t.co/cG9TMrqUoG

[@edwin 🌵📈 @edwin]

@vikati and I analyzed 2,430 Claude Code repo decisions.

Claude Code never picked AWS or GCP for deployment.

If agents are writing the first version of new projects, they’re influencing which tools get adopted at scale.

[@edwin 🌵📈 @edwin]

1/ AI is changing how brands appear in search. Try searching "best acoustic guitar under $500" in Google, ChatGPT, and Perplexity. You'll get three completely different sets of recommendations citing different sources

[@edwin 🌵📈 @edwin]

Speaking of AI bias & @AOC's comments, we did a bunch of tests and @azure, @awscloud, @clarifai & @IBMWatson all think @maddow is a man https://t.co/puBMmU7uT3

[Alex Vikati @vikati]

We just released a new Tether dataset that covers every transaction up to block 546906. Go to https://t.co/r6pRmCy4Bi if you are interested in doing primary Tether data analysis.

[Alex Vikati @vikati]

I just published Tether vs Other Stablecoins https://t.co/1n9m0Nv2uV