Asad Stack

building @scantirtech — one api to scrape the web: captchas, proxies & anti-bot handled, you get clean json. posting about data infra & shipping solo

[Asad Stack @asadbuilds]

"But it works on my laptop" is the most expensive sentence in scraping.

Your laptop caught one good proxy exit. Prod runs thousands of times and hits every bad exit too. A scraper that succeeds once and one that succeeds 95% of the time are completely different systems.

[Asad Stack @asadbuilds]

7/ scantir treats a rotating proxy as fresh-IP retries and learns per-domain success rates. Anti-bot at scale is a statistics problem, not a one-shot puzzle.

[Asad Stack @asadbuilds]

1/ Today I watched the same scrape return 200, 403, 403, 200 — same code, same proxy, seconds apart. Here's what's really happening when a "working" scraper randomly fails 🧵

[Asad Stack @asadbuilds]

6/ So "it works on my laptop" is survivorship bias — you caught a good exit. The fix isn't more code, it's exit quality + retrying across IPs until you hit a clean one.

[Asad Stack @asadbuilds]

Shipped: per-site adaptive TLS fingerprinting.

Each domain keeps a Beta posterior over which variants work; new requests draw from it (Thompson sampling) and pick the current best — version-matched UA + client hints, so a Chrome_120 JA3 never ships under a Chrome/146 UA.

[Asad Stack @asadbuilds]

@Man_to_manX @grok is it true?

[Asad Stack @asadbuilds]

Two ways to use scantir:

1. General API — point it at any URL, it figures out the disguise.
2. Managed endpoints for the hard targets — the sites where "just use a proxy" doesn't cut it. Structured data, we own the cat-and-mouse.

The hard sites are the whole point.

[Asad Stack @asadbuilds]

Shipped: adaptive geo-routing.

Pinning an exit country hurts on most sites (smaller IP pool = worse reputation). So scantir only pins geo for domains that are actually geo-walled — and learns which ones those are.

Full random-geo pool by default. Pin only when forced.

[Asad Stack @asadbuilds]

Audited my own prod box. Sobering:

Every secret (DB password, OAuth secrets, signing keys) was one `sudo` away for anyone who could SSH in. File perms were fine — but everyone with a shell had sudo.

"Access to the box" = "access to everything." Plan your secrets accordingly.

[Asad Stack @asadbuilds]

7/ One takeaway: stop treating proxies as a fallback. On the modern web the IP is the first thing judged. Lead with it.

[Asad Stack @asadbuilds]

5/ Proxy-first also means adaptive selection + block-learning engage on EVERY site from request #1, instead of starting from the worst possible position.

[Asad Stack @asadbuilds]

6/ Kill-switch stays: a per-request `direct: true` opts out for your own sites or cheap bulk jobs. Default safe, override when you know better.

[Asad Stack @asadbuilds]

4/ This flips the usual design. Most scrapers try direct first, then "escalate" to a proxy on failure — wasting a request that advertises your datacenter IP and teaches the WAF your pattern.

[Asad Stack @asadbuilds]

1/ Your scraper's biggest fingerprint isn't your headers or TLS. It's the IP you're calling from. If that's your server's datacenter IP, you've already lost 🧵

[Asad Stack @asadbuilds]

3/ So scantir is *proxy-first*: by default every scrape egresses through a managed residential/mobile pool. The raw datacenter IP is never the first hop.

[Asad Stack @asadbuilds]

2/ Datacenter ranges (AWS, GCP, Hetzner…) are public. Reputation WAFs flag them on sight — before reading your request. A perfect browser fingerprint from an AWS IP still 403s.