@IsaNikoumanesh You kind of nailed it 🤓 I wonder if @KorayGubur shares the same opinion that we can recall it as Bill Slawski's legacy evolution or not.
⚠️ But be careful! A 10-second delay means 8640 requests per day.
For a site with 500K pages, it would take AhrefsBot 57 days to crawl the entire website non-stop. That's why some crawlers refuse to let the 'crawl-delay' slow them down!
💡 Unofficial but effective: The 'crawl-delay' directive.
📝 In robots.txt, you can specify the minimum delay between requests for a specific bot:
User-agent: [Bot]
Crawl-Delay: [Value in seconds]
Most crawlers don't follow it. But @ahrefs does.
💬you know others? reply here.
💡How to join the https://t.co/bXWBXwhRWQ?
Registration is limited to invitation links.
You can use my invitation link:
https://t.co/cqgqxtZpU9
⚠️ #SEO:
They forgot to add "noindex" to /invite/ directory.
😁 so SEO folks are welcomed by default:
🔎 site:https://t.co/ALfLIiBuWY
Also, the page's HTML includes the date in <time> element.
<time title="July 21, 2015 12:09AM" datetime="2015-07-21T00:09:27+00:00">July 2015</time>
@JohnMu@dannysullivan
Are there guidelines for discussion sites URL-wise?
Isn't a URL a weaker source of information than HTML?
💡#SEO: May Google prefer data extracted from URL over schema?
Look at the date; it's "Aug 10, 1980" 😮
The Internet was born in 1983!
query: daedra hearts elder scrolls online
❓How did this happen? ... continue reading 🧵
h/t @darth_na
The schema defined the creation date, as shown in the screenshot.
Somehow, Google chose to extract the date from a URL segment that was meant to be a discussion ID.
🔗 .../discussion/198008/...
That's how 198008 became August, 1980.
Set and enforce an aspirational personal hourly rate. If fixing a problem will save less than your hourly rate, ignore it. If outsourcing a task will cost less than your hourly rate, outsource it.
Great video explanation of Google's newest "see in the dark" ML algorithm by @twominutepapers.
If you haven't seen it yet: it's the most impressive demo you'll see this month. Or you'll get your money back! 😉
https://t.co/8YtCg03d9b
💡#SEO
robots.txt and status codes like 404 and 410 don't waste the crawl budget.
Because GoogleBot gets back nothing but just status codes.
Watch out for soft 404; Cause pages affected skip indexing and waste crawl budget as well.
ℹ️ SOTR / @methode
missed @JohnMu; w r u?