Scott

Chicago Saluki focused on digital marketing, investing, & sports

[Scott @scottrandel]

The great blog collapse… https://t.co/hF3Q71QmNX

[Pedro Dias @pedrodias]

Cloudflare just reshaped how AI crawlers get classified — and SEO teams need to pay attention before Sept 15.

On its second "Content Independence Day," Cloudflare moved away from the blunt "block all AI bots" model toward a behaviour-based taxonomy. Instead of asking "is this AI?", the question is now what is the bot doing, what does it store, and how does it reshare your content?

The three use cases every site owner can now control (yes, including the Free tier):

☛ Search: indexing your content to answer queries later; the behavior that drives referral traffic back to you. Allowed by default.
☛ Agent: real-time bots acting on a human's behalf (ChatGPT-User, Gemini/Claude driving Chrome).
☛ Training: crawling to permanently absorb your content into a model.

The change with the biggest SEO implications ☛ new defaults land September 15, 2026.

On pages that display ads, Training and Agent will be blocked by default while Search stays allowed (for new domains onboarding). More importantly: multi-purpose crawlers will now be blocked according to all their behaviors. That means if you block Training, combined crawlers like Googlebot, Applebot, and Bingbot get blocked too — because they crawl for both Search and Training.

You can opt out in Security settings before Sept 15 to protect your search visibility. Don't sleep on this one.

For AI-visibility work:
☛ A new use signal extends Content Signals in robots.txt — use=immediate (store nothing), use=reference (index, excerpt, link back — the new default), or use=full (summarize and reproduce). These are stated preferences, not hard blocks.
☛ Bots that reproduce content in full can't be Verified. Abusing signals costs a bot its Verified status.
☛ "Verified" no longer means "allowed by default" — it now just makes a bot allowable within its category.

Also worth knowing:
☛ BotBase — a new searchable directory (Enterprise Bot Management) showing every tracked bot, its classification, and a copyable detection ID for security rules.
☛ Transitive trust — an experiment using the RFC 7239 Forwarded header (e.g. Forwarded: for="openai";use="reference") so operator-level trust decisions hold even through layers of intermediaries.

The through-line: control is getting granular, and the era of "search and training are the same crawler" is ending. If you manage crawl access, audit your Training rules now so you don't accidentally deindex yourself in September.

[Scott @scottrandel]

What is Prompt Caching? Optimize LLM Latency with AI Transformers https://t.co/pnSlxaRYhV automated and explicit caching

[OpenAI Developers @OpenAIDevs]

Show Codex a workflow once. Reuse it as a skill.

Record & Replay lets you show Codex a recurring task, like filing an expense report or submitting a time-off request.

Codex turns that demo into an inspectable, editable skill.

You control when recording starts and stops.

[Scott @scottrandel]

Been thinking about capability catalogs for agents. Will be interesting to see how this framework evolves

[Google for Developers @googledevs]

Agents are part of a massive, interconnected ecosystem. But how do they find and trust each other across different platforms?

Today, we’re proud to announce the Agentic Resource Discovery (ARD), an open specification alongside industry partners (including Cisco, Databricks, GitHub, GoDaddy, Hugging Face, Microsoft, NVIDIA, Salesforce, ServiceNow, and Snowflake). ARD gives any agent a secure, decentralized way to discover and verify capabilities (like tools, skills, MCP servers, and other agents) anywhere on the web.

Read the full announcement and get started: https://t.co/1I9WsT0tyH

[Scott @scottrandel]

@mikefutia SEO

[Scott @scottrandel]

@MartijnSch @shannholmberg Thanks! Now I need to run some experiments to see how many more tokens these costs

[Logan Kilpatrick @OfficialLoganK]

In @GoogleAIStudio we are now making more than 1,200,000 apps a week (and growing) with more than 18,000,000 created since late February 🤯

The progress continues!!!

[Scott @scottrandel]

@mikefutia Claude

[Scott @scottrandel]

@mikefutia Ads

[Scott @scottrandel]

@buccocapital @LynAldenContact Snipd @snipd_app does something similar

[Scott @scottrandel]

@mikefutia SEO

[Scott @scottrandel]

@mikefutia Skill

[Scott @scottrandel]

@biancoresearch @LynAldenContact US is up 31%, what’s the delta for the other regions?

[Scott @scottrandel]

100% agree

[Aaron Levie @levie]

There’s a fundamental difference between taking an existing process and applying AI agents to it vs. taking a process from scratch and designing it from the ground up for AI agents. The gap we’re going to see will widen between the teams and companies that are able to do the latter instead of just the former.

In theory it would have been ideal for all the gains of AI to have come “for free”, but there are both clear constraints of AI (like getting the context right) and clear upsides (like being able to execute code and run in parallel) that the workflows themselves must be redesigned to take full advantage of this technology.

One of the biggest implications that will come into focus is that agents that can write and run code, and interact with any API, will lead to agents effectively being expert engineers applied to your business process.

So to some extent one of the biggest ways of reengineering a workflow is to ask yourself: what would you do if you had an infinite number of capable engineers write software for this process. What if those engineers wrote code to connect your disparate data sources, comb thorough any amount of unstructured data, automate your repeated tasks, connect your various systems together specific to your process, and so on.

Not every process has that upside, but there tons of tasks that we do every day across marketing, finance, operations, and even sales, where a programmer with infinite code writing and API access would be able to make something go far faster or produce way more output. The teams that start to think this way will start to operate entirely differently.

[Aaron Levie @levie]

There are some pretty wild downstream effects in a world with trillions of agents using the internet and software.

One very big one is what happens with agents with budgets and wallets. There are lots of business models that never ended up working out for the human-based internet that all of a sudden start to make economic sense in an agent-based internet.

Think of all the proprietary data and research that’s sitting out there right now behind a paywall that a human will never run into. Finance data, medical research, and so on. Most people won’t sign up for a $100 or $1000 subscription for information they need infrequently. The cost is too high. Equally, micropayments for this data rarely worked at scale because the volume was too low to matter.

However, now an agent can have a budget for a specific set of research it’s doing, and the agent might pay $0.1 or $1 to access it in a workflow. And now that data may be relevant in 1,000X’s more use-cases than it was before.

Similarly, there are many APIs and tools out there on the web that don’t make sense to have a subscription for, but now an agent may interact with for a specific exchange, and it could cost $.01 or $0.1 per transaction.

All of a sudden new kinds of software can get built and monetized that would have been uneconomical before. Some new form of commercial open source, essentially.

Obviously lots of infrastructure and agreement across the industry is needed for this -and getting discovered by the agent is going to be a whole new class of search and discovery problem- but there are so many potentially interesting new scenarios here.

[Scott @scottrandel]

“Small talk is a tiny, free and very possibly priceless reminder of our shared humanity.”

https://t.co/Gc0H5jrsRg

[Scott @scottrandel]

@LynAldenContact @HoweGeneration Fair to say it’s here. 2020-2040 or maybe 35 is probably the real crisis era

[Scott @scottrandel]

@mikefutia Meta