Salience

Know what Google and AI bots actually did on your site today.

[Salience @saliencecom]

We are live: https://t.co/CiSjhIgP4c

[Carl Hendy @carlhendy]

A $100m global ecommerce brand rebranded and replatformed. By May, the number of keywords ranking in Google's top 3 had fallen 69% and estimated organic traffic was down 72%.

@audits was brought in to review the migration and help with the brand's longer-term ecommerce SEO strategy.

As part of the investigation, we connected the site to @Saliencecom to see what was happening when Googlebot and other crawlers visited.

A few things stood out:

- Googlebot was fetching robots.txt around 5x more often than on a sister site.
- Googlebot crawl activity had halved in a month, while the sister site was being crawled 6.6x as often.
- 16.6% of requests were being rate limited (429), including roughly 1 in 5 homepage requests from clients identifying as Googlebot.
- Some important pages were returning 404s, while paginated category URLs were switching between 200 and 404 responses.
- AI crawlers making 1.8x more requests than Googlebot.

The logs helped us identify the symptoms, but we needed Google Search Console and live header checks to understand what was causing them.

That uncovered two issues:

- An A/B test was assigning Googlebot to different variants across crawls.
- robots.txt and the sitemap were being served uncached from a legacy backend, which was also setting three session cookies on every request.

The recommendations were fairly straightforward: bypass the A/B test for crawlers and serve robots.txt and the sitemap from the edge cache.

Six weeks after the first fixes in late May, keywords ranking in the top 3 were up 182% on the May low and estimated organic traffic had doubled. The biggest jump came in August, about 11 weeks after the fixes.

Since the May low, according to Ahrefs US data:

- Keywords ranking in Google's top 3 are up 28x.
- Keywords ranking in Google's top 10 are up 52x.
- Estimated organic traffic is up 9x.

Anyone who's worked on recovering a failed migration will know how long it can take Google to catch up, particularly when the problems have been around for months.

For me, the most useful part of combining Salience's server log data with Search Console was having enough evidence to explain what was happening, justify the recommendations and get the client's development resources allocated to fixing them.

That can often be the hardest part.

We've since built checks for all issues discovered into Salience and tested them against the site's logs.

A lot of what we build comes from real SEO projects like this, with the aim of making this level of analysis and monitoring accessible to everyone.

[Salience @saliencecom]

Server log analysis for SEO
https://t.co/ukDTWbFwkh

[Salience @saliencecom]

OpenAI shifted from live reads to crawling
https://t.co/Fo1rS63xFM

[Salience @saliencecom]

https://t.co/0MV8eByWyc

[Carl Hendy @carlhendy]

Most SEOs have never looked at their server logs.

We’re all talking about AI visibility, LLM citations and whether AI systems are using our content. But how many of us actually know which AI bots are visiting our websites?

Google Search Console, analytics and third-party crawlers are useful, but none gives you the complete picture of requests hitting your infrastructure.

Server and CDN logs show you which URLs were requested, by which user-agents, when, and what response they received. And with streaming, you can see that activity within seconds.

Some of the things worth looking at:

AI and LLMs

• Which pages GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and other AI crawlers are requesting, and how often.

• The difference between AI training crawlers, search crawlers and user-triggered fetches. They’re not all doing the same thing.

• Whether AI crawlers are being blocked, rate-limited or receiving 403s, 429s and 5xx errors.

• Which AI bots are genuine and which are scrapers pretending to be recognised crawlers.

• Which sections of your website AI crawlers visit most, and which important pages they haven’t requested.

• How AI crawler activity changes over time, particularly after publishing new content or making technical changes.

• How AI crawler activity compares with actual referral traffic from ChatGPT, Perplexity and other assistants.

Traditional technical #SEO

• Which important pages Googlebot is crawling, how frequently it returns and how quickly it discovers new content.

• Where crawl activity is being wasted on parameters, faceted navigation, redirects and low-value URLs.

• Pages Googlebot keeps requesting but Google hasn’t indexed, or indexed pages it hasn’t revisited.

• Intermittent 404s, 5xx errors and unexpected redirects that regular website crawls might miss.

• Whether Googlebot is discovering new URLs and following the correct redirects after a migration or release.

• Slow response times, rate limiting and other issues affecting search engine crawlers.

• Orphan pages that bots continue requesting despite no longer being linked internally.

• Broken external links that people are actually clicking, only to land on a 404.

Of course, an AI crawler requesting a page doesn’t mean the content was used for training or cited in an AI answer. But at least you know what the bots are actually doing, rather than guessing.

Most websites have access to valuable first-party data that rarely gets looked at.

@saliencecom

[Carl Hendy @carlhendy]

Saw Quentin Yacoub's findings on the rapid growth of ExaSearchBot across Adobe customer websites, so thought I’d compare them with what we're seeing in Salience's server logs.

Turns out we're seeing a 5,271% increase against its August baseline.

Comparing 12–18 October against average weekly crawler activity in August:

📈 ExaSearchBot: +5,271%
📈 OAI-SearchBot: +782%
📈 PerplexityBot: +339%

Worth remembering Exa started from a much smaller baseline, so this isn't a comparison of total traffic.

I was curious about what might be driving the increase, so had a look at who's actually using Exa.

Turns out they've got some pretty significant customers and integrations, including Firefox, Cursor, Cognition, HubSpot, Monday, OpenRouter and Google Cloud.

They also recently announced 2 billion requests a month and have been massively expanding their own search index.

I wonder how much of the crawler growth we're seeing is Exa building out and refreshing that index, particularly ahead of the Exa 3.0 launch.

We also saw a pullback in activity over the latest matched days, so it's certainly not consistent growth every week.

Still, interesting to see similar patterns across two independent datasets.

As more AI products rely on third-party search providers, understanding who's actually crawling your website is becoming rather important.

[Dennis Goedegebuure @TheNextCorner]

🧭 @carlhendy 's team at @saliencecom reviewed ~1.5 billion server requests, comparing approximate weekly snapshots from mid-August to mid-October. Verified ChatGPT-User requests, the fetches triggered by a user's question, fell by approximately 80%. Verified OAI-SearchBot requests, OpenAI's search crawler, more than doubled. AI referral traffic stayed small next to Google organic. It's consistent with ChatGPT fetching less live and crawling more in advance. It doesn't show why, or whether anyone saw a recommendation. The site count and mix aren't disclosed.

My take: I suspect OpenAI is building its own search index, and probably a smaller one. My hypothesis, an assumption and not a measured figure: ~20% of prompts drive ~80% of demand, so an index covering that head could answer most of it. An own index is faster than a fresh search, and most facts no longer change. Fewer live searches means infrastructure cost that can be redeployed, and the same crawl helps with training data.

This industry is moving so fast, can you keep up?
CC :@hellemans
https://t.co/sf2qyRVchd
https://t.co/8EMaNf3rUF

[Carl Hendy @carlhendy]

We've been reviewing around 1.5 billion server requests, so I thought I'd share a few observations.

There's been some interesting discussion recently about how much traffic AI assistants actually send to websites.

Worth checking out the research and perspectives from @Marie_Haynes , Kevin Indig, The SEO Works, @randfish and Michael King. Different datasets, methodologies and opinions, all worth reading.

Firstly, AI referral traffic remains small compared with Google organic. Our findings broadly agree with the other studies. However, server logs reveal another layer of activity that traditional analytics doesn't capture.

Some interesting changes we've observed since mid-August:

✳️ ChatGPT live reads: Verified ChatGPT-User requests have fallen by approximately 80%.

✳️ OpenAI crawling: Verified OAI-SearchBot requests have more than doubled over the same period.

✳️ Claude Code: We recorded approximately 26x growth in distinct sources accessing websites between June and September, predominantly technical documentation.

The change in ChatGPT's behaviour is particularly interesting. It's consistent with a shift towards crawling and indexing content in advance, rather than fetching pages during individual conversations.

Of course, an AI agent fetching a page doesn't mean someone saw a recommendation, let alone made a purchase. But that doesn't make the activity irrelevant either.

It highlights why referral traffic, agent activity and indexing need to be considered separately.

Interested to hear whether others analysing server logs are seeing similar patterns.

[Carl Hendy @carlhendy]

A good example of why we built https://t.co/376bN6V8S0 and why real-time server log monitoring matters.

Following the recent PHP2Shell vulnerability, we spotted suspicious scanning activity across several websites were monitoring.

Around 20% of monitored sites received exploit-shaped requests, with some activity starting before a patch was even available.

Most requests were blocked at the edge, but a small number of sites warranted further investigation. We've already contacted the affected clients.

Its a useful reminder that server logs reveal activity you wont find in Google Analytics or Search Console, including requests blocked before they ever reach your application.

Security wasn't the primary reason we built Salience, but its another practical benefit of making first-party server data accessible and actionable.

[Carl Hendy @carlhendy]

Salience is now live: connect your website for free and start measuring first-party data that most teams never see, let alone act on.

Your server and CDN logs show what is actually reaching your site, what gets requested, how often crawlers return, what your infrastructure serves them and where things go wrong.

Built alongside my co-founders @ndredding and @channel5 , Salience is designed to make that raw log data genuinely useful and actionable in real time.

That includes seeing how search engines and AI crawlers such as ChatGPT and Claude interact with your site: which pages they visit, how frequently they return, what they request and whether they encounter errors.

We originally built it off the back of problems we were solving for clients, particularly around crawling, indexing, and AI visibility across products, collections and editorial content, and large marketplace listing websites while giving teams access to a layer of insight they’d rarely had before.

It’s designed for a broad range of sites, from Shopify and BigCommerce stores through to publishers, marketplaces and larger businesses running custom technology stacks.

It’s also useful beyond #SEO and search, with insights for development, content and security teams, whether you’re in-house, an agency or a consultant.

We’ve already had some great feedback from beta users, so thank you to everyone who has tested it so far.

We’d love more of it: what’s useful, what’s missing and what could be better.

There’s a working demo to explore, or you can connect a production site for free and see what your own data tells you.

Please share it with anyone on your team who might find it useful.

Visit: https://t.co/0YhHLhSzzb for more detail.