Crawl Central

The place to be for Crawl information and Data Discovery

[Crawl Central @crawlcentral]

The weekend is finally upon us!

Have a good couple of days off everyone!

[Crawl Central @crawlcentral]

Do we release on a Friday?! Of course we release on a Friday!

[Crawl Central @crawlcentral]

Distributed Crawls also have great fault tolerance if you've architected the platform correctly, allowing you to cope with failure without having to start from scratch again.

[Crawl Central @crawlcentral]

Happy friday folks. Hopefully you've had a great week!

[Crawl Central @crawlcentral]

What is your favourite programming language and why?

We enjoy Java and other JVM languages, they provide the most cross platform support, but we're also big fans of Python 😍

What about if your webcrawl was language agnostic?

[Crawl Central @crawlcentral]

Sometimes its not about crawling the competitors, its about crawling your own sites and making the results available to your own staff. If you're a company with many intranet sites Webcrawling meets Enterprise Search.

[Crawl Central @crawlcentral]

How do you productionalize a basic BeautifulSoup web crawl campaign?

Get in touch, we can help!

[Crawl Central @crawlcentral]

Proxy support
Multiple browser support
Header rotation
User interaction emulation

We have the bases covered. Get in touch for crawl help.

[Crawl Central @crawlcentral]

We'll have a hosted Crawl Central demo coming soon! We're looking forward to all the feedback. In the mean time, if you need crawls or support, get in contact and we'll be happy to help.

[Crawl Central @crawlcentral]

Did you know a lot of criminals hide their criminality in plain sight? This is how #DARPA leveraged web crawling to help track down criminal groups and organizations.

[Crawl Central @crawlcentral]

Taking Selenium Grid 4 for a spin. How cool is the new UI and the various updates?

Also the new Helm chart! https://t.co/HTK57CV0Kv

[Crawl Central @crawlcentral]

Question for all our crawl nerds.

What cool use cases do you use webscrapes and web data extraction for?

[Crawl Central @crawlcentral]

We also wrote a second scripting language for Selenium, slightly different and a work in progress. This one is called magnesium script, feel free to give it a go and get involved!
https://t.co/g9mFxJTnYS

[Crawl Central @crawlcentral]

If you want to nerd out over crawling strategies our @jrz813 is your boy!

[Crawl Central @crawlcentral]

What do you use to extract information after a Crawl? We're huge @ApacheTika fans over here!

[Crawl Central @crawlcentral]

Morning all! 😵

[Crawl Central @crawlcentral]

Want to know more about how to build scalable crawls? Get in touch!

[Crawl Central @crawlcentral]

Price analysis & competitor analysis projects can be greatly simplified using Selenium to interact with the target sites. Give Selenium Scripter a whirl: https://t.co/sDqf3cESfH

[Crawl Central @crawlcentral]

What's you're favourite cloud data service and why?

Can be anything and doesn't have to be super complicated. We'll start - S3, easy storage of Crawl results for post processing and analysis.

[Crawl Central @crawlcentral]

Happy Monday!