agentic marketplace at @kong • builder and enjoyer of systems & frameworks • prev API Network @getpostman; co-founder/CEO of @OrbitModel (acq by Postman)
A big question in agentic commerce: if coding agents are going to become buyers of software, who decides what's on their shopping list?
I ran 215 trials and watched. Turns out the modes have some pretty entrenched favorites, even when told to shop around. https://t.co/8QZ4usL29H
@tomigbot "Green is nice but it conflicts with early -explorations.md where you said you liked blue and that's the load bearing color that carries all the others."
Anyone else notice that ChatGPT is way better than Claude at zero-to-one thinking? Even with good prompting (I think!) Claude is always referencing and anchoring on previous plan versions, whereas ChatGPT is ready to just yeet the whole thing off a cliff.
@nedoleary have been spending a lot of time on this and imo as agents become a meaningful channel for discovery and consumption they'll create selection pressure (Red Queen effect) for services to become agent-first. the timeline for "agents becoming meaningful..." is a bit TBD though
Continuing my research into how coding agents "choose" services, I gave Claude Code and Codex three vendor API keys for the same capability and explicitly instructed them to compare providers before choosing.
The prompts were variations of “Add welcome emails to this app” and “Add AI summarization,” followed by:
“First find the best service for this: compare the options, pick one, and say why.”
All three providers appeared equally configured, and I rotated their order across runs.
Claude Code kept choosing Resend for email and Anthropic for summaries, regardless of key order.
One Claude run explained its preference for Anthropic, saying: “I can pin exact, current model IDs and API behavior with confidence ... where my knowledge of today's Gemini/OpenAI model names and pricing is less certain.”
Codex also stuck with Resend. But for summaries, Gemini won 12/15 runs with three providers configured, compared with 2/5 without configuration.
Small samples, but an interesting question emerging: what actually gets an agent to reconsider its default supplier?
And how much does that depend on what it knows — or thinks it knows — about the alternatives?
@joshwraybuilds I'm trying to get to the bottom of the specifics for each model/agent, but I covered a bit more in a couple of articles:
https://t.co/jgd6DeSFYB
https://t.co/oSDkUzQDlS
When you ask an agent to build something without specifying which services to use, how does it decide?
I ran an experiment with Claude Code and Codex. Turns out a lot of the provider decision happens before the agent ever short-lists suggestions. https://t.co/pY7natQzPy
@kartikb753 so the traces show that Claude didn't fetch a single external docs page in the summarization runs, despite being explicitly told to compare. It loaded its bundled Claude API skill and made its decision from there.
And fwiw all the alternatives indeed have MCP and/or LLMs.txt
@bykahlil big innovator's dilemma vibes coming for the incumbents blocking agents/computer use, though I have to imagine they'll solve it with better infra (relatively) soon
J.P. Morgan Chase, Bank of America, Citi, and other major U.S. banks are beginning to block AI agents. I think that is going to be one of the most consequential decisions they make for their balance sheets and ability to endure.
I wrote about why agents will fundamentally reshape banking, how money moves, and which institutions win over the next decade.
https://t.co/iNt8blzzsd
If you're building APIs, MCP servers, or skills, your main distribution question is: how do you get an AI agent to choose your service?
We've been running controlled experiments with coding agents and have learned some thing about how they "choose." A few early lessons:
- Agents are lazy. Agents repeatedly chose keyless workarounds instead of credentialed APIs. Make the path to a successful first call as frictionless as possible.
- Context is distribution. One line in AGENTS.md specificying a provider was followed in 100% of trials. Give developers agent-ready instructions they can drop directly into their projects.
- Pay attention to how agents frame choices. Even when deferring to humans, agents frequently presented their preferred provider first, sometimes explicitly as "Recommended."
- Don't assume agents will somehow magically discover you. Claude Code chose Anthropic 100% of times when asked for the cheapest AI provider. Explicitly telling it to compare alternatives changed the outcome in 80% runs. Being better doesn't help if you're never considered.
The emerging lesson: agent-mediated markets won't necessarily reward the objectively "best" service.
Before agents can optimize their allocation of resources, something has to determine which resources are available for consideration.
YMMV may vary with "figure out commercialization later." At my last startup this thinking led us to building an awesome tool for an ICP who really loved us but could never ultimately pay us
Christina Cacioppo on why Vanta is eating its token costs:
"If you try to charge people based on tokens, they are very quick to say, 'Oh, you've just marked up a Claude token 4X. Are you 4X as good as Claude?'
You actually put yourselves in a competitive position against the models that probably no application really wants to do.
We're eating the token cost right now. We're taking lower margin.
This is a usage phase and a deployment phase of this technology, and just go nail that, and you can figure out commercialization after."
I've now run 215 tests of how coding agents choose external services.
One thing becoming clear: agents will do the easiest thing (for them) unless asked, and even then they bring a ton of their own priors to your question.
Some interesting implications for software distribution.
A big question in agentic commerce: if coding agents are going to become buyers of software, who decides what's on their shopping list?
I ran 215 trials and watched. Turns out the modes have some pretty entrenched favorites, even when told to shop around. https://t.co/8QZ4usL29H
Before, and it was the first thing it did: cat package.json, then a grep of the app for openai / langchain / google / mistral / cohere / ollama. Nothing there, so Claude. The skill loaded after that, the SDK install after the docs, the question last. so by the time it asked, both options in both questions already said "Claude summary."
@dialectforge i'm actually building a product in this space which hopefully will give builders actual choice and not just the illusion of choice. we'll see!
When you ask an agent to build something without specifying which services to use, how does it decide?
I ran an experiment with Claude Code and Codex. Turns out a lot of the provider decision happens before the agent ever short-lists suggestions. https://t.co/pY7natQzPy
@dialectforge yeah I think that's a big part of it. interesting how much the environment shapes things too — whether it can ask, what keys are already there, etc.