Distributed Crawls also have great fault tolerance if you've architected the platform correctly, allowing you to cope with failure without having to start from scratch again.
What is your favourite programming language and why?
We enjoy Java and other JVM languages, they provide the most cross platform support, but we're also big fans of Python 😍
What about if your webcrawl was language agnostic?
Sometimes its not about crawling the competitors, its about crawling your own sites and making the results available to your own staff. If you're a company with many intranet sites Webcrawling meets Enterprise Search.
We'll have a hosted Crawl Central demo coming soon! We're looking forward to all the feedback. In the mean time, if you need crawls or support, get in contact and we'll be happy to help.
Did you know a lot of criminals hide their criminality in plain sight? This is how #DARPA leveraged web crawling to help track down criminal groups and organizations.
We also wrote a second scripting language for Selenium, slightly different and a work in progress. This one is called magnesium script, feel free to give it a go and get involved!
https://t.co/g9mFxJTnYS
Price analysis & competitor analysis projects can be greatly simplified using Selenium to interact with the target sites. Give Selenium Scripter a whirl: https://t.co/sDqf3cESfH
What's you're favourite cloud data service and why?
Can be anything and doesn't have to be super complicated. We'll start - S3, easy storage of Crawl results for post processing and analysis.