@domainsgeneral Our database is updated several times a day. However, if and when new links are found depends on many factors. Basically: For technical and legal reasons, we cannot permanently crawl every single page, so we crawl different domains differently often and deeply.
@iphigenie @Doopie_NL This is definitely important, crawling such pages is of course without any value for us. So we are always grateful for hints where something doesn't work well, so we can improve our crawling algorithm.
@Doopie_NL @iphigenie The same offer to you: If you think we should improve anything, please send me the affected domain and we will check the logs in detail. In case of blocking, please use robots.txt, as this works regardless of the IP address we are using. I would naturally prefer an investigation.
@iphigenie Of course you are free to block our crawler partially or completely via robots.txt. If you want us to investigate if there is an error on our site, just tell us the domain and we will look at the crawling log in detail.