audits.com

SEO Audits Trusted by Brands, Investors, and In-house Teams.

[Carl Hendy @carlhendy]

Oops!

[Carl Hendy @carlhendy]

A $100m global ecommerce brand rebranded and replatformed. By May, the number of keywords ranking in Google's top 3 had fallen 69% and estimated organic traffic was down 72%.

@audits was brought in to review the migration and help with the brand's longer-term ecommerce SEO strategy.

As part of the investigation, we connected the site to @Saliencecom to see what was happening when Googlebot and other crawlers visited.

A few things stood out:

- Googlebot was fetching robots.txt around 5x more often than on a sister site.
- Googlebot crawl activity had halved in a month, while the sister site was being crawled 6.6x as often.
- 16.6% of requests were being rate limited (429), including roughly 1 in 5 homepage requests from clients identifying as Googlebot.
- Some important pages were returning 404s, while paginated category URLs were switching between 200 and 404 responses.
- AI crawlers making 1.8x more requests than Googlebot.

The logs helped us identify the symptoms, but we needed Google Search Console and live header checks to understand what was causing them.

That uncovered two issues:

- An A/B test was assigning Googlebot to different variants across crawls.
- robots.txt and the sitemap were being served uncached from a legacy backend, which was also setting three session cookies on every request.

The recommendations were fairly straightforward: bypass the A/B test for crawlers and serve robots.txt and the sitemap from the edge cache.

Six weeks after the first fixes in late May, keywords ranking in the top 3 were up 182% on the May low and estimated organic traffic had doubled. The biggest jump came in August, about 11 weeks after the fixes.

Since the May low, according to Ahrefs US data:

- Keywords ranking in Google's top 3 are up 28x.
- Keywords ranking in Google's top 10 are up 52x.
- Estimated organic traffic is up 9x.

Anyone who's worked on recovering a failed migration will know how long it can take Google to catch up, particularly when the problems have been around for months.

For me, the most useful part of combining Salience's server log data with Search Console was having enough evidence to explain what was happening, justify the recommendations and get the client's development resources allocated to fixing them.

That can often be the hardest part.

We've since built checks for all issues discovered into Salience and tested them against the site's logs.

A lot of what we build comes from real SEO projects like this, with the aim of making this level of analysis and monitoring accessible to everyone.

[Carl Hendy @carlhendy]

A reminder of just how badly an ecommerce replatforming can go wrong.

In 2021, Adorn Cosmetics was celebrating its move to Adobe Commerce. Five years later, 7NEWS is reporting on the serious fallout the founder says followed.

The founder, Briony Kennedy, has publicly raised concerns about Adobe and the agency involved in recommending and implementing the move. She's now reportedly pursuing legal action over what happened.

I've had a quick look at the current website and spotted some fairly obvious issues from a search / SEO perspective. But without knowing what was scoped, implemented or changed over the years, it would be unfair to comment on who did what or where things went wrong.

It's a reminder of just how much is at stake when replatforming or rebranding a business. Technical SEO, conversion, customer experience and the wider commercial impact all need careful consideration.

I'd argue it's even harder today than it was five years ago. We're no longer just working around Google's crawling and indexing timeframes. We've now got LLMs, AI crawlers and agentic search to accommodate as well.

Every migration carries risks, regardless of the platform. But there are also significant opportunities when it's planned and executed properly.

[Carl Hendy @carlhendy]

Most SEOs have never looked at their server logs.

We’re all talking about AI visibility, LLM citations and whether AI systems are using our content. But how many of us actually know which AI bots are visiting our websites?

Google Search Console, analytics and third-party crawlers are useful, but none gives you the complete picture of requests hitting your infrastructure.

Server and CDN logs show you which URLs were requested, by which user-agents, when, and what response they received. And with streaming, you can see that activity within seconds.

Some of the things worth looking at:

AI and LLMs

• Which pages GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and other AI crawlers are requesting, and how often.

• The difference between AI training crawlers, search crawlers and user-triggered fetches. They’re not all doing the same thing.

• Whether AI crawlers are being blocked, rate-limited or receiving 403s, 429s and 5xx errors.

• Which AI bots are genuine and which are scrapers pretending to be recognised crawlers.

• Which sections of your website AI crawlers visit most, and which important pages they haven’t requested.

• How AI crawler activity changes over time, particularly after publishing new content or making technical changes.

• How AI crawler activity compares with actual referral traffic from ChatGPT, Perplexity and other assistants.

Traditional technical #SEO

• Which important pages Googlebot is crawling, how frequently it returns and how quickly it discovers new content.

• Where crawl activity is being wasted on parameters, faceted navigation, redirects and low-value URLs.

• Pages Googlebot keeps requesting but Google hasn’t indexed, or indexed pages it hasn’t revisited.

• Intermittent 404s, 5xx errors and unexpected redirects that regular website crawls might miss.

• Whether Googlebot is discovering new URLs and following the correct redirects after a migration or release.

• Slow response times, rate limiting and other issues affecting search engine crawlers.

• Orphan pages that bots continue requesting despite no longer being linked internally.

• Broken external links that people are actually clicking, only to land on a 404.

Of course, an AI crawler requesting a page doesn’t mean the content was used for training or cited in an AI answer. But at least you know what the bots are actually doing, rather than guessing.

Most websites have access to valuable first-party data that rarely gets looked at.

@saliencecom

[Carl Hendy @carlhendy]

We've been reviewing around 1.5 billion server requests, so I thought I'd share a few observations.

There's been some interesting discussion recently about how much traffic AI assistants actually send to websites.

Worth checking out the research and perspectives from @Marie_Haynes , Kevin Indig, The SEO Works, @randfish and Michael King. Different datasets, methodologies and opinions, all worth reading.

Firstly, AI referral traffic remains small compared with Google organic. Our findings broadly agree with the other studies. However, server logs reveal another layer of activity that traditional analytics doesn't capture.

Some interesting changes we've observed since mid-August:

✳️ ChatGPT live reads: Verified ChatGPT-User requests have fallen by approximately 80%.

✳️ OpenAI crawling: Verified OAI-SearchBot requests have more than doubled over the same period.

✳️ Claude Code: We recorded approximately 26x growth in distinct sources accessing websites between June and September, predominantly technical documentation.

The change in ChatGPT's behaviour is particularly interesting. It's consistent with a shift towards crawling and indexing content in advance, rather than fetching pages during individual conversations.

Of course, an AI agent fetching a page doesn't mean someone saw a recommendation, let alone made a purchase. But that doesn't make the activity irrelevant either.

It highlights why referral traffic, agent activity and indexing need to be considered separately.

Interested to hear whether others analysing server logs are seeing similar patterns.

[Carl Hendy @carlhendy]

A good example of why we built https://t.co/376bN6V8S0 and why real-time server log monitoring matters.

Following the recent PHP2Shell vulnerability, we spotted suspicious scanning activity across several websites were monitoring.

Around 20% of monitored sites received exploit-shaped requests, with some activity starting before a patch was even available.

Most requests were blocked at the edge, but a small number of sites warranted further investigation. We've already contacted the affected clients.

Its a useful reminder that server logs reveal activity you wont find in Google Analytics or Search Console, including requests blocked before they ever reach your application.

Security wasn't the primary reason we built Salience, but its another practical benefit of making first-party server data accessible and actionable.

[Carl Hendy @carlhendy]

Early morning call with a US client.

“Your work is exquisite.”

Not a word I usually associate with dry technical SEO, but nice to hear.

[Carl Hendy @carlhendy]

https://t.co/o6UIYufWzd

[Carl Hendy @carlhendy]

Salience is now live: connect your website for free and start measuring first-party data that most teams never see, let alone act on.

Your server and CDN logs show what is actually reaching your site, what gets requested, how often crawlers return, what your infrastructure serves them and where things go wrong.

Built alongside my co-founders @ndredding and @channel5 , Salience is designed to make that raw log data genuinely useful and actionable in real time.

That includes seeing how search engines and AI crawlers such as ChatGPT and Claude interact with your site: which pages they visit, how frequently they return, what they request and whether they encounter errors.

We originally built it off the back of problems we were solving for clients, particularly around crawling, indexing, and AI visibility across products, collections and editorial content, and large marketplace listing websites while giving teams access to a layer of insight they’d rarely had before.

It’s designed for a broad range of sites, from Shopify and BigCommerce stores through to publishers, marketplaces and larger businesses running custom technology stacks.

It’s also useful beyond #SEO and search, with insights for development, content and security teams, whether you’re in-house, an agency or a consultant.

We’ve already had some great feedback from beta users, so thank you to everyone who has tested it so far.

We’d love more of it: what’s useful, what’s missing and what could be better.

There’s a working demo to explore, or you can connect a production site for free and see what your own data tells you.

Please share it with anyone on your team who might find it useful.

Visit: https://t.co/0YhHLhSzzb for more detail.

[Carl Hendy @carlhendy]

Technical SEO is so back!

In the early days of Google, a lot of SEO was about making sure search engines could actually crawl, understand and index your website and get links, get so many links!

Then, through the 2010s and into the early 2020s, the focus shifted much more towards content, digital PR (links), UX, brand, entities and trust.

Technical SEO never went away, obviously, but on a lot of established websites it became less of the problem.

That seems to have changed again.

I’m seeing more technical SEO issues this year than I have in the last few years combined.

A lot of that is down to how websites, apps, and web assets are being built and teams operate now.

We’ve moved from fairly rigid CMS platforms to headless builds (often not required), custom frameworks, internal tools, dynamically generated pages and now a growing amount of vibe-coded websites, tools and content assets.

That gives teams the autonomy they need and removes developer bottlenecks. 

It also gives them far more ways to break things.

- JavaScript rendering issues.
- multiple frameworks.
- subdomain party central.
- Content that exists for users but not for crawlers.
- Infinite URLs.
- Cloaking.
- Wrong status codes.
- Internal links that aren’t really links.
- Content hidden behind interactions.
- Caching problems.
- Server header responses.
- Resources accidentally blocked.
- Staging sites indexed.
- Production sites accidentally noindexed.

You’ve got Googlebot, Bingbot, AI search crawlers, training crawlers and retrieval bots, all behaving slightly differently.

Some clients want certain AI bots to have access and others blocked.

Some bots handle JavaScript better than others.

Some follow the same controls. Others have their own.

So after 20+ years of SEO, we’ve somehow ended up asking many of the same questions again.

- Can the bot (all of them) access it?
- Can it render it?
- Can it understand what it’s looking at?
- Is it seeing the version we actually want it to see?
- Have we accidentally blocked it?

The tech and internet has changed.

A lot of the technical #SEO problems really haven’t.

[Carl Hendy @carlhendy]

Once you learn that SEO is contextual, a lot of advice starts to fall apart.

What works in one search vertical may not work in another.

What works for one website may not work for yours.

What works for a competitor may be completely wrong for your site.

That’s why SEO blueprints, frameworks and cheat sheets rarely scale cleanly across established websites.

The context is the strategy.

[Carl Hendy @carlhendy]

Built a useful AI monitoring tool.

https://t.co/IkgjKjbnSp

[Carl Hendy @carlhendy]

Felt cute, might delete later.

A bit of fun at https://t.co/ArQZKJxMBg.

[Carl Hendy @carlhendy]

Early days, but the recovery from this failed website migration is finally starting to get going. Hopefully, I’ll be able to talk about this one publicly one day. It’s a banger.

[Carl Hendy @carlhendy]

Updated my free https://t.co/zx7WS0Pd1R tool.

Compare 5 ecommerce competitors across 30+ signals, including Google Shopping, Core Web Vitals, Lighthouse, Knowledge Graph, trust checks, review momentum and more.

Still 100% free. No signup. All public Google data.

[Carl Hendy @carlhendy]

I've rebuilt my free Knowledge Graph Search tool. Search any brand or person to see what Google knows about them, whether they have a verified Google profile, and how Wikipedia, Wikidata and Common Crawl see them. You can even compare two entities side by side.

[Carl Hendy @carlhendy]

2/2 Undefeated SEO Quiz Champion. Retiring at the top.

[Carl Hendy @carlhendy]

@Cloudflare's new AEO Visibility product feels like Similarweb for AI discovery: tracking brand mentions, citations, prominence and share of voice across models, then adding crawler and referral data from the site itself.

[Carl Hendy @carlhendy]

Nature is healing.

[Carl Hendy @carlhendy]

Nine years ago, I predicted Google would kill both cache: and site:. I was right about cache:, just years too early 😅

Now I’m doubling down: site: will be next, partly because LLMs and scrapers use it to access Google results.