Screaming Frog

Run by Founder Dan Sharp of Screaming Frog, a UK search marketing agency & developer of the SEO Spider. Gooner. SEO. & Philanthropist.

[Screaming Frog @screamingfrog]

Just released @screamingfrog SEO Spider v.24:

https://t.co/wSFQFFrUf6

Includes -

🤖Screaming Frog MCP

🧑‍🤝‍🧑Auto Compare Crawls

📊View Crawl Changes in Email Notifications

📧Send Crawl Export Attachments by Email

🚫Find Uncrawlable Links

& lots more! Enjoy.

[Screaming Frog @screamingfrog]

@DanielTexasLira You can just input the XML Sitemap Index file, and it will crawl all the XML Sitemaps in it (and all the URLs in them). We wouldn't crawl nested sitemaps though (ie sitemap index to other sitemap index to sitemaps).

[Tomek Rudzki @TomekRudzki]

Full article https://t.co/rolboLEvNR

[Screaming Frog @screamingfrog]

@TomekRudzki Welcome back, mate! 🔥

[Screaming Frog @screamingfrog]

@DanFoland Sorry Dan, slow to reply! No, there isn't a way to do a post-analysis currently. We have it on the 'todo', but at the moment search/extraction is all at runtime of the crawl.

[Screaming Frog @screamingfrog]

No, that wouldn’t make sense alone. Fundamentally under the covers the only difference is list mode has crawl depth set to ‘0’. You can have diff configs between list and spider mode don’t forget though (i.e user-agent, speed etc) - so perhaps there is something else in play in your settings. If you’ve added your IP to an allowlist and you’re getting 429s though, that would suggest it’s not working as it should obv!

[Screaming Frog @screamingfrog]

Just released a small update to 24.1 of the @screamingfrog SEO Spider - https://t.co/5jdr39oEvR.

Includes various updates to the SEO Spider MCP, small improvements and bug fixes.

[Screaming Frog @screamingfrog]

Introducing Search Generative AI performance reports in Search Console - https://t.co/QGscP1blSG from @googlesearchc

[Screaming Frog @screamingfrog]

New tutorial on 'How To Set Up Crawl Email Notifications & Export Attachments' in @screamingfrog - https://t.co/M3CQJ1YKOj

[Screaming Frog @screamingfrog]

@HenningSeo Cheers, Henning. We've got an item on the 'todo' to allow users to set the number of auto retries for no response/connection timeouts as well (similar to what we have now for 5XX).

[Screaming Frog @screamingfrog]

@richvolleruk Congrats on the first post, nice write up! 👍

[Screaming Frog @screamingfrog]

@PropelDMS Check out the API list in our docs (https://t.co/uggeuD5Xfr), or within the MCP config in the tool ('View Tools'). The sf_crawl you can start a crawl with an optional config profile!

[Screaming Frog @screamingfrog]

Just released @screamingfrog SEO Spider v.24:

https://t.co/wSFQFFrUf6

Includes -

🤖Screaming Frog MCP

🧑‍🤝‍🧑Auto Compare Crawls

📊View Crawl Changes in Email Notifications

📧Send Crawl Export Attachments by Email

🚫Find Uncrawlable Links

& lots more! Enjoy.

[Screaming Frog @screamingfrog]

@PropelDMS Pop through to support and we can explain next steps - https://t.co/keqqW5mNkm

[Screaming Frog @screamingfrog]

@RednetBhushan @chris_nectiv You can pop through details to our support and explain what mode you’re trying and what’s happened - https://t.co/keqqW5mNkm

[Screaming Frog @screamingfrog]

@bree_sharp Thanks, Bree!

[Screaming Frog @screamingfrog]

@nathanojaokomo Thanks, Nathan! 😂

[Screaming Frog @screamingfrog]

@dodgejd Cheers, Jason!

[Screaming Frog @screamingfrog]

@Hobo_Web Thank you, mate - too kind! 👍

[Screaming Frog @screamingfrog]

@RednetBhushan @chris_nectiv Install version 24, and then follow the user guide (if you install the MCP extension before v.24, it won't work) - https://t.co/uggeuD5Xfr

[Screaming Frog @screamingfrog]

@carlos_darko Just wrote a little FAQ on it as well here - https://t.co/69Ac7LJb4A

[Screaming Frog @screamingfrog]

@carlos_darko Make sure you install version 24 first, then install the MCP extension (some folk are doing it before!). Restart Claude and it should work. If not, [email protected] with the dets pls.