Ian Kerins

Bootstrapping @ScrapeOps. I talk about building & marketing a SAAS business. 🚀

[Ian Kerins @IanJKerins]

1/🧵
Something strange is happening in web scraping…
Proxy prices have dropped 70–90%, infra is faster than ever…

Yet scraping costs are exploding.

Why?

Welcome to the Proxy Paradox - where cheaper proxies somehow make scraping more expensive. 🤯

[Ian Kerins @IanJKerins]

That's the thread. Asked for tips on keeping 100+ sources under control, Pierluigi's answer fit in one line:

"Standardize, Document, Best practices. And then create all the skills about processes and your documentation, so that an AI could help you."

The bigger shift: AI cuts the effort of building and maintaining scrapers. The work moves to what it doesn't solve on its own, which is checking completeness, costing the whole pipeline and knowing who will pay for the data.

Full write-up: https://t.co/1JU7grx5gZ

Original AMA thread: https://t.co/oNGhzYOoxI

LinkedIn breakdown: https://t.co/IkXqO65B7E

When a scraper breaks, what do you check first: the IP, the selector, or the tooling?

[Ian Kerins @IanJKerins]

A couple of months to set up the AI.

A couple of minutes per scraper to maintain.

That's how Pierluigi Vinciguerra(@pigivinciguerra) describes running AI on his team's scrapers in AMA #14 in r/WebScrapingInsider. The minutes are an average for standard websites: the hour a fix takes, spread over the months when nothing breaks. The months went into writing down best practices, standards, workflows and tool docs first.

Pierluigi has been in web scraping for 12+ years. He runs @webscrapingclub newsletter and is co-founder and CTO of Data Boutique, a marketplace for web-scraped data.

What he says is hard now: anti-bot bypass, knowing an extraction is complete, and the tradeoff between reliability and cost.

Six insights and three bonus answers. 🧵

[Ian Kerins @IanJKerins]

+ 3/3: starting solo? Specialise in a stack, not a vertical.

Someone asked whether a solo starter should niche around one vertical's data or one technical stack across many niches. His whole answer:

"I think focus on your stack, not a vertical"

For a new marketplace seller picking a first dataset, he suggested "finding datasets of websites related to the ones listed and not yet available". The other route is to create an account and wait for buyers to ask for websites, though "you should be able to answer quickly."

It's a one-line answer with no reasoning attached, so read it as his opinion, not a rule.

[Ian Kerins @IanJKerins]

@_alexspring, Founder of Driver(@driverdotdev), is joining r/WebScrapingInsider for a live AMA on running real Chrome on physical devices instead of VMs.

🕟 Thu, Nov 5, 4:30 PM EST

https://t.co/CaIcgV6Zok

can real hardware beat anti-bot systems? 🖥️

[Ian Kerins @IanJKerins]

They use historical SERP data as a baseline.

For each keyword, they track which domains consistently appear in the top 10. If fewer than 50% of those usual domains show up in a new SERP, they flag it as potentially poisoned.

They also monitor how quickly those domains return. Nozzle found that most poisoning episodes are temporary, often lasting just one SERP.

Essentially, they're detecting anomalies by comparing today's SERP against its historical fingerprint, not by knowing what Google should show.

[Ian Kerins @IanJKerins]

A block is the easy failure. At least it tells you something went wrong.

Nozzle's new research reports that Google is serving poisoned search results to automated requests: complete pages, no CAPTCHA, no error, nothing in the HTML that flags the problem.

In its desktop sample, 22.6% of SERPs since September 13 were classified as poisoned on page 1 or 2, peaking at 43.5%.

That's the lead story in Web Scraping Insider #11 🧵

[Ian Kerins @IanJKerins]

Full issue: https://t.co/MYCE9Tcgg5

Join the 5,000: https://t.co/5GZX6TV0Ol

The new home for the Insider: https://t.co/dh8EEZzDSY

How do you catch bad data that comes back as a clean 200: comparing against history, validating by hand, or not catching it yet?

[Ian Kerins @IanJKerins]

None of this replaces the boring work.

A 200 response and a healthy success rate can both hide bad data. Validate what comes back, benchmark providers on your own targets, and measure cost-per-validated-payload.