Web Scraping Proxies

Match IP type to target difficulty

Send soft targets to datacenter proxies at a tenth of the cost, and reserve residential for the endpoints that actually block you. A tiered router in front of your crawler typically cuts proxy spend by 60–80%.

Fingerprints beyond the IP

Cloudflare, DataDome, PerimeterX and Kasada score TLS (JA3/JA4), HTTP/2 frame order, and JS runtime signals. A perfect residential IP with a Python-requests TLS fingerprint still gets a 403.

Rotation and session strategy

Rotate per request only for stateless endpoints. For paginated or authenticated flows, hold a sticky session and retire the IP once it starts returning challenges.

Measure the right metric

Track successful parsed records per dollar, not requests per second. A cheap pool with a 40% block rate and three retries costs more than a premium pool at 97%.

Building a tiered proxy router

The single biggest cost lever in a scraping stack is not the price per gigabyte — it is which requests you send through expensive IPs. A tiered router classifies every target domain into soft, medium and hard, then sends soft traffic through dedicated datacenter proxies, medium traffic through ISP static residential, and only the hard tail through rotating residential or an unlocker endpoint.

In production this is a lookup table keyed by domain plus a feedback loop: when a domain's rolling success rate drops below a threshold, it is promoted a tier; when it stays above 99% for a week on a cheaper tier, it is demoted. Teams that adopt tiering typically cut proxy spend by 60–80% without losing coverage, because the overwhelming majority of URLs in any crawl come from tolerant endpoints that never needed a residential IP.

Track cost per successfully parsed record, not cost per gigabyte. A residential pool at $8/GB that returns clean HTML on the first attempt is cheaper than a $1/GB pool that needs six retries and still fails a fifth of the time.

Rotation, sessions and concurrency

Rotation is not a setting you turn up. Stateless list pages tolerate a new IP per request; anything behind a login, a cart or a multi-step filter needs a sticky session held for the life of that flow. Most modern gateways expose both on the same host — a rotating port and a session-ID suffix on the username — so the crawler can choose per request type.

Concurrency should be expressed per exit IP, not globally. Two hundred threads spread across ten thousand IPs looks like ordinary background noise; the same two hundred threads on fifty IPs looks like an attack. Rate-limit per IP and per domain, add jitter, and back off exponentially on the first 429 rather than the tenth.

Retries deserve a policy of their own: retry on a different exit, in a different subnet, after a delay, and cap attempts. Blind immediate retries on the same IP are the fastest way to burn a subnet across an entire provider pool.

Fingerprints matter as much as IPs

Modern anti-bot systems fingerprint the TLS handshake (JA3/JA4), the HTTP/2 frame ordering, and the header casing your client emits — long before they look at the IP. A perfect residential proxy driving a default Python requests session still announces itself as a script on the first byte.

Use a client whose fingerprint matches the browser you claim to be: curl-impersonate, a real Chromium via Playwright, or a managed unlocker API that maintains the impersonation for you. Keep header order, Accept-Language and timezone consistent with the proxy's exit country — a German IP sending en-US headers at 03:00 local time is a trivial signal to catch.

When success rates collapse on a target that used to work, test the fingerprint before you blame the pool. Nine times out of ten a vendor shipped a new detection rule, not a bad IP.

Legal, ethical and robots considerations

Collecting publicly accessible data is broadly accepted in most jurisdictions, but that is not a blanket licence. Respect robots.txt where it expresses a crawl-rate preference, avoid logged-in or paywalled content you have no right to, never scrape personal data without a lawful basis, and keep a documented retention policy.

On the supply side, ask your provider how the residential pool is sourced, whether peers gave informed consent, and whether the company passes SOC 2 or ISO 27001. Ethical sourcing is now a procurement requirement for enterprise buyers, and an unlicensed pool can make a downstream dataset unsellable.

How to choose a provider for web scraping

Start from the workload, not the price list. For web scraping the recommended class is rotating residential · datacenter for soft targets, with a very high traffic profile — that pair alone eliminates most of the market. Then score the shortlist on four things: measured success rate against your own targets, geographic and ASN granularity, session control, and the billing model.

Every credible provider offers a trial or a small pay-as-you-go tranche. Run the same hundred requests through each candidate, from the same client, against your real targets, and compare success rate and median latency. Vendor benchmarks are run on empty pools at favourable hours; yours will not be.

Finally, read the refund and expiry terms. Non-expiring traffic, prorated refunds and the ability to pause a plan are worth more than a 10% discount when workloads are seasonal.

Pricing and budgeting for web scraping

Three billing models dominate. Per-gigabyte pricing suits residential and mobile pools and rewards efficient clients that block images and analytics. Per-port or per-IP pricing suits static ISP, dedicated datacenter and mobile ports, and is the only sane model when one identity must hold one address. Per-request pricing, used by unlockers and scraper APIs, transfers the block risk to the vendor and is the cheapest option on genuinely hard targets.

Model the total, not the headline. Add failed-request overhead, the bandwidth your client wastes on assets you never parse, and the engineering time spent maintaining parsers. Efficiency work — response compression, asset blocking, conditional requests, aggressive caching — routinely halves a bill without changing provider.

Commit annually only after a full month of production traffic. Volume discounts are real, but so is the cost of being locked into a pool that turns out to be weak on your key geos.

Rotation, sessions and geo targeting in practice

Rotation policy should follow state. Stateless requests can take a new exit every time; anything with a login, a cart or a multi-step form needs a sticky session that outlives the flow. Most gateways support both concurrently, selected by port or by a session token in the username.

Geo targeting granularity matters more than pool size for most buyers. Country-level is table stakes; city, region, ZIP and ASN or carrier targeting are what let you reproduce a specific user's experience. Verify the targeting actually works by resolving the exit IP's geolocation against two independent databases — advertised coverage and delivered coverage diverge more often than vendors admit.

Common mistakes in web scraping setups

Buying on pool size alone. Headline IP counts are unaudited and include long-inactive peers; a smaller, well-maintained pool with good ASN diversity outperforms a bigger stale one on every hard target.

Rotating when you should be sticky, and sticky when you should be rotating. Both fail loudly — one as account-takeover challenges, the other as rate-limit blocks.

Ignoring the client fingerprint. A default HTTP library behind a premium residential IP is still detectable in the first packet.

No observability. Without per-target success rates, per-IP error codes and a cost-per-record metric, you cannot tell a provider problem from a code problem, and you will keep switching vendors instead of fixing the crawler.

Measuring success and knowing when to switch

Instrument four metrics from day one: success rate per target domain, median and p95 latency per exit country, cost per successful record, and ban half-life — how long a fresh IP survives on a given target. Together they tell you whether a degradation is the pool, the target's new defences, or your own code.

Set a review cadence. Re-benchmark the top three providers quarterly with the same script; pools shift as peers churn and vendors buy or lose supply. A provider that led a year ago is not automatically the right answer today, and switching is cheap when your crawler treats the gateway as configuration rather than architecture.